2023-11-08T08:00:37.449 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T08:00:37.450 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_neesingh@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} --name neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi --archive /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824 -- /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/orig.config.yaml 2023-11-08T08:00:37.493 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T08:00:37.579 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824 branch: wip-neesingh-testing-54557 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} email: neesingh@redhat.com first_in_suite: false job_id: '7451824' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-neesingh-testing-54557 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: b820560f30d310ab2570761b9a294523225d6998 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b820560f30d310ab2570761b9a294523225d6998 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-neesingh-testing-54557 sha1: b820560f30d310ab2570761b9a294523225d6998 owner: scheduled_neesingh@teuthology priority: 10 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5135 sha1: b820560f30d310ab2570761b9a294523225d6998 sleep_before_teardown: 0 suite: fs suite_branch: wip-neesingh-testing-54557 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b820560f30d310ab2570761b9a294523225d6998 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmWrVyaQLvu7kH/xOHecmPZs0Wv4Vnk8ebDAlklT4ioAMKeecUfXKCBKWoONiWEWSMBVYZ7doSnrlZcsf0LePE= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP75L6TR03Z2dV6ScTobCTEsiEwEJDOeIRunLkcy5G1avJ+xdQ9cy9Bds1buAVhxI7AAzWZJ//guXa7HXaHSMi0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-08_07:51:45 tube: smithi user: neesingh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 2023-11-08T08:00:37.580 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa; will attempt to use it 2023-11-08T08:00:37.581 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa/tasks 2023-11-08T08:00:37.581 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-08T08:00:37.583 INFO:teuthology.task.internal:Checking packages... 2023-11-08T08:00:37.609 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b820560f30d310ab2570761b9a294523225d6998' 2023-11-08T08:00:37.609 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-08T08:00:37.610 INFO:teuthology.packaging:ref: None 2023-11-08T08:00:37.611 INFO:teuthology.packaging:tag: None 2023-11-08T08:00:37.611 INFO:teuthology.packaging:branch: wip-neesingh-testing-54557 2023-11-08T08:00:37.612 INFO:teuthology.packaging:sha1: b820560f30d310ab2570761b9a294523225d6998 2023-11-08T08:00:37.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-neesingh-testing-54557 2023-11-08T08:00:37.922 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2698.gb820560f 2023-11-08T08:00:37.922 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-08T08:00:37.929 INFO:teuthology.task.internal:no buildpackages task found 2023-11-08T08:00:37.929 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-08T08:00:37.942 INFO:teuthology.task.internal:Saving configuration 2023-11-08T08:00:37.959 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-08T08:00:37.973 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T08:00:37.998 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 07:55:31.821565', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmWrVyaQLvu7kH/xOHecmPZs0Wv4Vnk8ebDAlklT4ioAMKeecUfXKCBKWoONiWEWSMBVYZ7doSnrlZcsf0LePE='} 2023-11-08T08:00:38.018 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 07:55:31.823139', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP75L6TR03Z2dV6ScTobCTEsiEwEJDOeIRunLkcy5G1avJ+xdQ9cy9Bds1buAVhxI7AAzWZJ//guXa7HXaHSMi0='} 2023-11-08T08:00:38.018 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-08T08:00:38.027 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-08T08:00:38.028 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-08T08:00:38.028 INFO:teuthology.run_tasks:Running task console_log... 2023-11-08T08:00:38.101 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb35b6e7c10>, signals=[15]) 2023-11-08T08:00:38.102 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-08T08:00:38.113 INFO:teuthology.task.internal:Opening connections... 2023-11-08T08:00:38.114 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:00:38.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:00:38.185 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:00:38.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:00:38.255 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-08T08:00:38.267 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2023-11-08T08:00:38.290 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2023-11-08T08:00:38.290 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2023-11-08T08:00:38.352 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2023-11-08T08:00:38.352 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8" 2023-11-08T08:00:38.352 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2023-11-08T08:00:38.353 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2023-11-08T08:00:38.353 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8" 2023-11-08T08:00:38.353 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8" 2023-11-08T08:00:38.353 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T08:00:38.353 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2023-11-08T08:00:38.354 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T08:00:38.354 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2023-11-08T08:00:38.354 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T08:00:38.354 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T08:00:38.355 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T08:00:38.355 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2023-11-08T08:00:38.385 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2023-11-08T08:00:38.407 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2023-11-08T08:00:38.408 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2023-11-08T08:00:38.470 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2023-11-08T08:00:38.470 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="8" 2023-11-08T08:00:38.470 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2023-11-08T08:00:38.470 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2023-11-08T08:00:38.471 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="8" 2023-11-08T08:00:38.471 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el8" 2023-11-08T08:00:38.471 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T08:00:38.471 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2023-11-08T08:00:38.472 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T08:00:38.472 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2023-11-08T08:00:38.472 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T08:00:38.472 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T08:00:38.472 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T08:00:38.473 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-11-08T08:00:38.493 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-08T08:00:38.502 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-08T08:00:38.545 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-08T08:00:38.545 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2023-11-08T08:00:38.548 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2023-11-08T08:00:38.569 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-08T08:00:38.580 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-08T08:00:38.580 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2023-11-08T08:00:38.609 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2023-11-08T08:00:38.639 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-08T08:00:38.700 INFO:teuthology.run_tasks:Running task kernel... 2023-11-08T08:00:38.719 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-08T08:00:38.720 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-08T08:00:38.721 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-08T08:00:38.722 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-08T08:00:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:00:38.742 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2023-11-08T08:00:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:00:38.744 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-11-08T08:00:38.803 INFO:teuthology.orchestra.run.smithi129.stdout:4.18.0-489.el8.x86_64 2023-11-08T08:00:38.803 INFO:teuthology.task.kernel:Running kernel on smithi129: 4.18.0-489.el8.x86_64 2023-11-08T08:00:38.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2023-11-08T08:00:38.806 INFO:teuthology.orchestra.run.smithi116.stdout:4.18.0-489.el8.x86_64 2023-11-08T08:00:38.806 INFO:teuthology.task.kernel:Running kernel on smithi116: 4.18.0-489.el8.x86_64 2023-11-08T08:00:38.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2023-11-08T08:00:43.667 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - AppStream 15 MB/s | 34 MB 00:02 2023-11-08T08:00:43.818 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - AppStream 14 MB/s | 34 MB 00:02 2023-11-08T08:00:51.384 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - BaseOS 34 MB/s | 53 MB 00:01 2023-11-08T08:00:52.568 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - BaseOS 21 MB/s | 53 MB 00:02 2023-11-08T08:01:00.542 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - Extras 11 kB/s | 18 kB 00:01 2023-11-08T08:01:00.950 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2023-11-08T08:01:01.261 INFO:teuthology.orchestra.run.smithi129.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-11-08T08:01:01.801 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - Extras 11 kB/s | 18 kB 00:01 2023-11-08T08:01:02.418 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-11-08T08:01:02.686 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-11-08T08:01:04.819 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 16 MB 00:03 2023-11-08T08:01:05.870 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 16 MB 00:03 2023-11-08T08:01:08.683 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 279 kB/s | 24 kB 00:00 2023-11-08T08:01:09.753 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 431 kB/s | 24 kB 00:00 2023-11-08T08:01:12.310 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-08T08:01:12.311 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-08T08:01:12.584 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-11-08T08:01:12.586 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-11-08T08:01:12.586 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2023-11-08T08:01:12.586 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-11-08T08:01:12.586 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-11-08T08:01:12.587 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-08T08:01:12.587 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-11-08T08:01:12.587 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-08T08:01:12.588 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-08T08:01:12.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:01:12.588 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-11-08T08:01:12.588 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2023-11-08T08:01:12.588 INFO:teuthology.orchestra.run.smithi129.stdout:Install 3 Packages 2023-11-08T08:01:12.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:01:12.589 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 89 M 2023-11-08T08:01:12.589 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 96 M 2023-11-08T08:01:12.589 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-11-08T08:01:13.321 INFO:teuthology.orchestra.run.smithi129.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2023-11-08T08:01:13.352 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-08T08:01:13.353 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-08T08:01:13.630 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-08T08:01:13.630 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-08T08:01:13.631 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2023-11-08T08:01:13.631 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-08T08:01:13.631 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-08T08:01:13.631 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-08T08:01:13.631 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-08T08:01:13.632 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-08T08:01:13.632 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-08T08:01:13.632 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:01:13.632 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-08T08:01:13.632 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-08T08:01:13.633 INFO:teuthology.orchestra.run.smithi116.stdout:Install 3 Packages 2023-11-08T08:01:13.633 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:01:13.633 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 89 M 2023-11-08T08:01:13.634 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 96 M 2023-11-08T08:01:13.634 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-08T08:01:14.246 INFO:teuthology.orchestra.run.smithi129.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 23 MB/s | 36 MB 00:01 2023-11-08T08:01:14.563 INFO:teuthology.orchestra.run.smithi116.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2023-11-08T08:01:14.980 INFO:teuthology.orchestra.run.smithi129.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-11-08T08:01:14.980 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-11-08T08:01:14.980 INFO:teuthology.orchestra.run.smithi129.stdout:Total 37 MB/s | 89 MB 00:02 2023-11-08T08:01:15.660 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-11-08T08:01:15.688 INFO:teuthology.orchestra.run.smithi116.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 24 MB/s | 43 MB 00:01 2023-11-08T08:01:16.005 INFO:teuthology.orchestra.run.smithi116.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2023-11-08T08:01:16.005 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-08T08:01:16.005 INFO:teuthology.orchestra.run.smithi116.stdout:Total 38 MB/s | 89 MB 00:02 2023-11-08T08:01:16.088 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-11-08T08:01:16.088 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-11-08T08:01:16.681 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-08T08:01:17.090 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-08T08:01:17.090 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-08T08:01:17.328 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-11-08T08:01:17.349 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-11-08T08:01:18.312 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-08T08:01:18.331 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-08T08:01:20.127 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-11-08T08:01:20.632 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T08:01:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-08T08:01:21.681 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T08:01:24.193 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T08:01:24.557 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T08:01:25.246 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T08:01:25.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T08:01:27.984 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T08:01:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:29.024 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T08:01:29.320 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:50.131 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:50.155 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:50.512 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:50.512 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-08T08:01:50.513 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-08T08:01:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:51.204 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:51.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:51.251 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:01:51.251 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-11-08T08:01:51.252 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-08T08:01:51.252 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-08T08:01:51.252 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:01:51.252 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-11-08T08:01:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-08T08:01:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-08T08:01:52.077 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T08:01:52.271 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T08:01:52.271 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:01:52.271 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-08T08:01:52.273 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-08T08:01:52.273 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-08T08:01:52.274 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:01:52.274 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-08T08:01:52.291 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T08:01:52.292 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.292 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.292 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-08T08:01:52.293 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-08T08:01:52.293 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-11-08T08:01:52.293 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2023-11-08T08:01:52.294 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T08:01:52.393 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T08:01:52.504 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T08:01:52.505 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2023-11-08T08:01:52.539 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2023-11-08T08:01:52.539 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2023-11-08T08:01:52.539 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.02 2023-11-08T08:01:52.540 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 129.el8 2023-11-08T08:01:52.540 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2023-11-08T08:01:52.540 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-08T08:01:52.540 INFO:teuthology.orchestra.run.smithi129.stdout:Group : System Environment/Base 2023-11-08T08:01:52.540 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 9265586 2023-11-08T08:01:52.541 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2023-11-08T08:01:52.541 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-08T08:01:52.541 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-08T08:01:52.541 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-08T08:01:52.541 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-08T08:01:52.542 INFO:teuthology.orchestra.run.smithi129.stdout:Relocations : (not relocatable) 2023-11-08T08:01:52.542 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : CentOS BuildSystem 2023-11-08T08:01:52.542 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2023-11-08T08:01:52.542 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-08T08:01:52.543 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2023-11-08T08:01:52.543 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2023-11-08T08:01:52.543 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:01:52.543 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-08T08:01:52.543 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-08T08:01:52.543 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-08T08:01:52.544 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2023-11-08T08:01:52.544 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:01:52.544 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2023-11-08T08:01:52.545 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-08T08:01:52.546 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-08T08:01:52.652 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T08:01:52.652 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.652 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.653 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-08T08:01:52.653 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-08T08:01:52.653 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-08T08:01:52.654 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2023-11-08T08:01:52.654 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T08:01:52.865 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T08:01:52.865 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2023-11-08T08:01:52.899 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2023-11-08T08:01:52.899 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2023-11-08T08:01:52.900 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.02 2023-11-08T08:01:52.900 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 129.el8 2023-11-08T08:01:52.900 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2023-11-08T08:01:52.900 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-08T08:01:52.900 INFO:teuthology.orchestra.run.smithi116.stdout:Group : System Environment/Base 2023-11-08T08:01:52.901 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 9265586 2023-11-08T08:01:52.901 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2023-11-08T08:01:52.901 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-08T08:01:52.901 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-08T08:01:52.902 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-08T08:01:52.902 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-08T08:01:52.902 INFO:teuthology.orchestra.run.smithi116.stdout:Relocations : (not relocatable) 2023-11-08T08:01:52.902 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : CentOS BuildSystem 2023-11-08T08:01:52.902 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2023-11-08T08:01:52.903 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-08T08:01:52.903 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2023-11-08T08:01:52.903 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2023-11-08T08:01:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:01:52.903 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-08T08:01:52.904 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-08T08:01:52.904 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-08T08:01:52.904 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2023-11-08T08:01:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:01:52.904 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2023-11-08T08:01:52.905 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-08T08:01:52.906 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 4.18.0-521.el8.x86_64 2023-11-08T08:01:52.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-08T08:01:53.989 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2023-11-08T08:01:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2023-11-08T08:01:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:done 2023-11-08T08:01:56.144 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-11-08T08:01:56.171 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.tnJord2RTX 2023-11-08T08:01:56.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tnJord2RTX 2023-11-08T08:01:56.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.tnJord2RTX 2023-11-08T08:01:56.392 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.tnJord2RTX is 6KB 2023-11-08T08:01:56.440 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.tnJord2RTX 2023-11-08T08:01:56.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2023-11-08T08:01:56.497 INFO:teuthology.orchestra.run.smithi129.stderr:done 2023-11-08T08:01:56.499 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-11-08T08:01:56.526 INFO:teuthology.orchestra.run.smithi116.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-08T08:01:56.526 INFO:teuthology.orchestra.run.smithi116.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-08T08:01:56.526 INFO:teuthology.orchestra.run.smithi116.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-08T08:01:56.527 INFO:teuthology.orchestra.run.smithi116.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-08T08:01:56.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-08T08:01:56.535 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.sewDMCi7xz 2023-11-08T08:01:56.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sewDMCi7xz 2023-11-08T08:01:56.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.sewDMCi7xz 2023-11-08T08:01:56.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2023-11-08T08:01:56.754 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.sewDMCi7xz is 6KB 2023-11-08T08:01:56.801 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.sewDMCi7xz 2023-11-08T08:01:56.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2023-11-08T08:01:56.897 INFO:teuthology.orchestra.run.smithi129.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-08T08:01:56.898 INFO:teuthology.orchestra.run.smithi129.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-08T08:01:56.898 INFO:teuthology.orchestra.run.smithi129.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-08T08:01:56.898 INFO:teuthology.orchestra.run.smithi129.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-08T08:01:56.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-08T08:01:57.063 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2023-11-08T08:02:26.718 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-08T08:02:26.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:02:26.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:02:27.066 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-08T08:02:27.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-11-08T08:02:27.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:02:45.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-11-08T08:02:54.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:02:54.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:02:57.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-11-08T08:03:01.627 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-11-08T08:03:09.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:03:09.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:03:10.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-11-08T08:03:10.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:03:27.743 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-11-08T08:03:29.018 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-11-08T08:03:41.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-11-08T08:03:41.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:03:41.475 DEBUG:teuthology.orchestra.run.smithi129:> true 2023-11-08T08:03:41.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-11-08T08:03:41.712 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-521.el8.x86_64"... 2023-11-08T08:03:41.712 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2023-11-08T08:03:41.773 INFO:teuthology.orchestra.run.smithi129.stdout:4.18.0-521.el8.x86_64 2023-11-08T08:03:41.773 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-08T08:03:41.773 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-08T08:03:41.774 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-08T08:03:42.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:03:42.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:03:42.775 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-08T08:03:42.775 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-08T08:03:43.108 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-11-08T08:03:43.359 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:03:43.359 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-521.el8.x86_64"... 2023-11-08T08:03:43.360 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-11-08T08:03:43.376 INFO:teuthology.orchestra.run.smithi116.stdout:4.18.0-521.el8.x86_64 2023-11-08T08:03:43.376 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-08T08:03:43.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-08T08:03:43.377 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-08T08:03:43.470 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2023-11-08T08:03:43.501 DEBUG:teuthology.parallel:result is None 2023-11-08T08:03:44.378 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-08T08:03:44.378 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-08T08:03:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2023-11-08T08:03:44.972 DEBUG:teuthology.parallel:result is None 2023-11-08T08:03:44.973 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-08T08:03:44.980 INFO:teuthology.task.internal:Creating test directory... 2023-11-08T08:03:44.980 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-08T08:03:45.023 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-08T08:03:45.395 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-08T08:03:45.407 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-08T08:03:45.430 INFO:teuthology.task.internal:Creating archive directory... 2023-11-08T08:03:45.431 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-08T08:03:45.457 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-08T08:03:45.858 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-08T08:03:45.867 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-08T08:03:45.867 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-08T08:03:45.872 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-08T08:03:45.906 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T08:03:45.910 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T08:03:45.919 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T08:03:45.921 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T08:03:45.925 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-08T08:03:45.937 INFO:teuthology.task.internal:Configuring sudo... 2023-11-08T08:03:45.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-08T08:03:45.960 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-08T08:03:45.990 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-08T08:03:46.002 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-08T08:03:46.003 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-08T08:03:46.025 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-08T08:03:46.048 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T08:03:46.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T08:03:46.220 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T08:03:46.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T08:03:46.307 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:03:46.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-08T08:03:46.373 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T08:03:46.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T08:03:46.475 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T08:03:46.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T08:03:46.604 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:03:46.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-08T08:03:46.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2023-11-08T08:03:46.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2023-11-08T08:03:46.719 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T08:03:46.790 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T08:03:47.249 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-08T08:03:47.257 INFO:teuthology.task.internal:Starting timer... 2023-11-08T08:03:47.257 INFO:teuthology.run_tasks:Running task pcp... 2023-11-08T08:03:47.272 INFO:teuthology.run_tasks:Running task selinux... 2023-11-08T08:03:47.283 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-08T08:03:47.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2023-11-08T08:03:47.365 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2023-11-08T08:03:47.367 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2023-11-08T08:03:47.451 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2023-11-08T08:03:47.453 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-08T08:03:47.453 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2023-11-08T08:03:47.480 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2023-11-08T08:03:47.481 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2023-11-08T08:03:47.509 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2023-11-08T08:03:47.509 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2023-11-08T08:03:47.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T08:03:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:03:47.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T08:03:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:03:47.577 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-08T08:03:47.577 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2023-11-08T08:03:47.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2023-11-08T08:03:47.644 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-08T08:03:47.655 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-08T08:03:47.655 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-08T08:03:47.671 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-08T08:03:47.672 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-08T08:11:06.097 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2023-11-08T08:11:06.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:11:06.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:11:06.212 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-11-08T08:11:06.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-08T08:11:06.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-11-08T08:11:06.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T08:11:06.309 DEBUG:teuthology.orchestra.run.smithi129:> true 2023-11-08T08:11:06.335 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-11-08T08:11:06.336 INFO:teuthology.run_tasks:Running task clock... 2023-11-08T08:11:06.350 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-08T08:11:06.351 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-08T08:11:06.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T08:11:06.354 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-08T08:11:06.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T08:11:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-08T08:11:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-08T08:11:06.408 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-08T08:11:06.422 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-08T08:11:06.471 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2023-11-08T08:11:06.482 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2023-11-08T08:11:06.490 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2023-11-08T08:11:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-08T08:11:06.501 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2023-11-08T08:11:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-08T08:11:06.515 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-08T08:11:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-08T08:11:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:bash: ntpq: command not found 2023-11-08T08:11:06.573 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T08:11:06.574 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-11-08T08:11:06.574 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.574 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.574 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.575 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.593 INFO:teuthology.orchestra.run.smithi129.stderr:bash: ntpq: command not found 2023-11-08T08:11:06.596 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T08:11:06.597 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2023-11-08T08:11:06.597 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.597 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.597 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.598 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:11:06.599 INFO:teuthology.run_tasks:Running task install... 2023-11-08T08:11:06.608 DEBUG:teuthology.task.install:project ceph 2023-11-08T08:11:06.608 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b820560f30d310ab2570761b9a294523225d6998'}} 2023-11-08T08:11:06.609 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b820560f30d310ab2570761b9a294523225d6998'} 2023-11-08T08:11:06.609 INFO:teuthology.task.install:Using flavor: default 2023-11-08T08:11:06.615 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-08T08:11:06.615 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-08T08:11:06.616 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b820560f30d310ab2570761b9a294523225d6998', 'tag': None, 'wait_for_package': False} 2023-11-08T08:11:06.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b820560f30d310ab2570761b9a294523225d6998 2023-11-08T08:11:06.619 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b820560f30d310ab2570761b9a294523225d6998', 'tag': None, 'wait_for_package': False} 2023-11-08T08:11:06.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b820560f30d310ab2570761b9a294523225d6998 2023-11-08T08:11:06.767 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/ 2023-11-08T08:11:06.768 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2698.gb820560f 2023-11-08T08:11:06.837 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/ 2023-11-08T08:11:06.837 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2698.gb820560f 2023-11-08T08:11:06.878 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-08T08:11:06.879 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:11:06.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-08T08:11:06.913 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-08T08:11:06.914 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b820560f30d310ab2570761b9a294523225d6998/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-08T08:11:06.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-08T08:11:07.008 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-54557/b820560f30d310ab2570761b9a294523225d6998/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-08T08:11:07.009 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:11:07.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-08T08:11:07.037 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-08T08:11:07.037 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b820560f30d310ab2570761b9a294523225d6998/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-08T08:11:07.059 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-08T08:11:07.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-08T08:11:07.125 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2023-11-08T08:11:07.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2023-11-08T08:11:07.181 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-08T08:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2023-11-08T08:11:07.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2023-11-08T08:11:07.393 INFO:teuthology.orchestra.run.smithi129.stdout:76 files removed 2023-11-08T08:11:07.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-08T08:11:07.514 INFO:teuthology.orchestra.run.smithi116.stdout:76 files removed 2023-11-08T08:11:07.541 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-08T08:11:08.203 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 214 kB/s | 80 kB 00:00 2023-11-08T08:11:08.246 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 264 kB/s | 80 kB 00:00 2023-11-08T08:11:08.504 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 36 kB/s | 9.4 kB 00:00 2023-11-08T08:11:08.605 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 29 kB/s | 9.4 kB 00:00 2023-11-08T08:11:08.796 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2023-11-08T08:11:08.964 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2023-11-08T08:11:10.330 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - AppStream 23 MB/s | 34 MB 00:01 2023-11-08T08:11:10.464 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - AppStream 23 MB/s | 34 MB 00:01 2023-11-08T08:11:18.214 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - BaseOS 33 MB/s | 53 MB 00:01 2023-11-08T08:11:19.131 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 53 MB 00:02 2023-11-08T08:12:01.179 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - Extras 515 B/s | 18 kB 00:35 2023-11-08T08:12:01.720 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-08T08:12:03.123 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 8 - PowerTools 5.3 MB/s | 6.6 MB 00:01 2023-11-08T08:12:04.558 INFO:teuthology.orchestra.run.smithi129.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-08T08:12:06.801 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2023-11-08T08:12:10.634 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 453 kB/s | 24 kB 00:00 2023-11-08T08:12:14.984 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-08T08:12:14.985 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-08T08:12:14.999 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2023-11-08T08:12:14.999 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2023-11-08T08:12:15.000 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2023-11-08T08:12:15.001 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2023-11-08T08:12:15.002 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-08T08:12:15.004 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-08T08:12:15.007 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-08T08:12:15.008 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-08T08:12:15.250 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2023-11-08T08:12:15.263 INFO:teuthology.orchestra.run.smithi129.stdout:==================================================================================================== 2023-11-08T08:12:15.263 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2023-11-08T08:12:15.264 INFO:teuthology.orchestra.run.smithi129.stdout:==================================================================================================== 2023-11-08T08:12:15.264 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2023-11-08T08:12:15.264 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2023-11-08T08:12:15.264 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:18.0.0-2698.gb820560f.el8 ceph 6.4 k 2023-11-08T08:12:15.265 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:18.0.0-2698.gb820560f.el8 ceph 5.4 M 2023-11-08T08:12:15.265 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:18.0.0-2698.gb820560f.el8 ceph 845 k 2023-11-08T08:12:15.265 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2698.gb820560f.el8 ceph 166 k 2023-11-08T08:12:15.265 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:18.0.0-2698.gb820560f.el8 ceph 1.5 M 2023-11-08T08:12:15.265 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 123 k 2023-11-08T08:12:15.266 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 1.7 M 2023-11-08T08:12:15.266 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 7.4 M 2023-11-08T08:12:15.266 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 47 k 2023-11-08T08:12:15.266 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:18.0.0-2698.gb820560f.el8 ceph 13 M 2023-11-08T08:12:15.266 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:18.0.0-2698.gb820560f.el8 ceph 47 M 2023-11-08T08:12:15.267 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 263 k 2023-11-08T08:12:15.267 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 83 k 2023-11-08T08:12:15.267 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:18.0.0-2698.gb820560f.el8 ceph 258 k 2023-11-08T08:12:15.267 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 18 k 2023-11-08T08:12:15.267 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 baseos 62 k 2023-11-08T08:12:15.268 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2023-11-08T08:12:15.268 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2023-11-08T08:12:15.268 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:18.0.0-2698.gb820560f.el8 ceph 30 k 2023-11-08T08:12:15.268 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 767 k 2023-11-08T08:12:15.268 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:18.0.0-2698.gb820560f.el8 ceph 139 k 2023-11-08T08:12:15.269 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2023-11-08T08:12:15.269 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.32.1-43.el8 baseos 100 k 2023-11-08T08:12:15.269 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:1.1.1k-9.el8 baseos 3.2 M 2023-11-08T08:12:15.269 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2023-11-08T08:12:15.269 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2023-11-08T08:12:15.270 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2023-11-08T08:12:15.270 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:18.0.0-2698.gb820560f.el8 ceph 190 k 2023-11-08T08:12:15.270 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:18.0.0-2698.gb820560f.el8 ceph 371 k 2023-11-08T08:12:15.270 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:18.0.0-2698.gb820560f.el8 ceph 364 k 2023-11-08T08:12:15.271 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:18.0.0-2698.gb820560f.el8 ceph 107 k 2023-11-08T08:12:15.271 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:18.0.0-2698.gb820560f.el8 ceph 94 k 2023-11-08T08:12:15.271 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:18.0.0-2698.gb820560f.el8 ceph 3.4 M 2023-11-08T08:12:15.271 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:18.0.0-2698.gb820560f.el8 ceph 179 k 2023-11-08T08:12:15.271 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2023-11-08T08:12:15.272 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2023-11-08T08:12:15.272 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-11.el8 baseos 144 k 2023-11-08T08:12:15.272 INFO:teuthology.orchestra.run.smithi129.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2023-11-08T08:12:15.272 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client x86_64 0.189-3.el8 baseos 76 k 2023-11-08T08:12:15.272 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf x86_64 0.189-3.el8 baseos 232 k 2023-11-08T08:12:15.273 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs x86_64 0.189-3.el8 baseos 303 k 2023-11-08T08:12:15.273 INFO:teuthology.orchestra.run.smithi129.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2023-11-08T08:12:15.273 INFO:teuthology.orchestra.run.smithi129.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2023-11-08T08:12:15.273 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2023-11-08T08:12:15.274 INFO:teuthology.orchestra.run.smithi129.stdout: libblkid x86_64 2.32.1-43.el8 baseos 220 k 2023-11-08T08:12:15.274 INFO:teuthology.orchestra.run.smithi129.stdout: libfdisk x86_64 2.32.1-43.el8 baseos 254 k 2023-11-08T08:12:15.274 INFO:teuthology.orchestra.run.smithi129.stdout: libmount x86_64 2.32.1-43.el8 baseos 236 k 2023-11-08T08:12:15.274 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 3.7 M 2023-11-08T08:12:15.274 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 3.6 M 2023-11-08T08:12:15.275 INFO:teuthology.orchestra.run.smithi129.stdout: libsmartcols x86_64 2.32.1-43.el8 baseos 179 k 2023-11-08T08:12:15.275 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid x86_64 2.32.1-43.el8 baseos 99 k 2023-11-08T08:12:15.275 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux x86_64 2.32.1-43.el8 baseos 2.5 M 2023-11-08T08:12:15.275 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-user x86_64 2.32.1-43.el8 baseos 103 k 2023-11-08T08:12:15.275 INFO:teuthology.orchestra.run.smithi129.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2023-11-08T08:12:15.276 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2023-11-08T08:12:15.276 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2023-11-08T08:12:15.276 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2023-11-08T08:12:15.276 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:18.0.0-2698.gb820560f.el8 ceph 22 M 2023-11-08T08:12:15.276 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 23 k 2023-11-08T08:12:15.277 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:18.0.0-2698.gb820560f.el8 ceph 2.3 M 2023-11-08T08:12:15.277 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 236 k 2023-11-08T08:12:15.277 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:18.0.0-2698.gb820560f.el8 ceph 4.6 M 2023-11-08T08:12:15.277 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:18.0.0-2698.gb820560f.el8 ceph 18 M 2023-11-08T08:12:15.277 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 15 k 2023-11-08T08:12:15.278 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:18.0.0-2698.gb820560f.el8 ceph 24 k 2023-11-08T08:12:15.278 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-08T08:12:15.278 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-08T08:12:15.278 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-08T08:12:15.279 INFO:teuthology.orchestra.run.smithi129.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2023-11-08T08:12:15.279 INFO:teuthology.orchestra.run.smithi129.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2023-11-08T08:12:15.279 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:18.0.0-2698.gb820560f.el8 ceph 137 k 2023-11-08T08:12:15.279 INFO:teuthology.orchestra.run.smithi129.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2023-11-08T08:12:15.279 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-08T08:12:15.280 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-08T08:12:15.280 INFO:teuthology.orchestra.run.smithi129.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2023-11-08T08:12:15.280 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-08T08:12:15.280 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-08T08:12:15.280 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 474 k 2023-11-08T08:12:15.281 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-08T08:12:15.281 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 6.5 M 2023-11-08T08:12:15.281 INFO:teuthology.orchestra.run.smithi129.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2023-11-08T08:12:15.281 INFO:teuthology.orchestra.run.smithi129.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2023-11-08T08:12:15.281 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-08T08:12:15.282 INFO:teuthology.orchestra.run.smithi129.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2023-11-08T08:12:15.282 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2023-11-08T08:12:15.282 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-08T08:12:15.282 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-08T08:12:15.282 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-08T08:12:15.283 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-08T08:12:15.283 INFO:teuthology.orchestra.run.smithi129.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2023-11-08T08:12:15.283 INFO:teuthology.orchestra.run.smithi129.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2023-11-08T08:12:15.283 INFO:teuthology.orchestra.run.smithi129.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2023-11-08T08:12:15.283 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-08T08:12:15.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-08T08:12:15.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-08T08:12:15.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-08T08:12:15.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-08T08:12:15.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:18.0.0-2698.gb820560f.el8 ceph 45 k 2023-11-08T08:12:15.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:18.0.0-2698.gb820560f.el8 ceph 117 k 2023-11-08T08:12:15.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-08T08:12:15.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-08T08:12:15.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-08T08:12:15.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-08T08:12:15.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-08T08:12:15.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-08T08:12:15.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-08T08:12:15.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-08T08:12:15.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-08T08:12:15.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-08T08:12:15.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-08T08:12:15.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-08T08:12:15.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-08T08:12:15.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-08T08:12:15.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-08T08:12:15.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-08T08:12:15.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-08T08:12:15.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-08T08:12:15.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-08T08:12:15.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-08T08:12:15.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-08T08:12:15.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-08T08:12:15.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-08T08:12:15.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-08T08:12:15.290 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-08T08:12:15.290 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-08T08:12:15.290 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-08T08:12:15.290 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-08T08:12:15.290 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-08T08:12:15.291 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-08T08:12:15.291 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-08T08:12:15.291 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-08T08:12:15.291 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-08T08:12:15.291 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-08T08:12:15.292 INFO:teuthology.orchestra.run.smithi129.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-08T08:12:15.292 INFO:teuthology.orchestra.run.smithi129.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-08T08:12:15.292 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-08T08:12:15.292 INFO:teuthology.orchestra.run.smithi129.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-08T08:12:15.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-08T08:12:15.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-08T08:12:15.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-08T08:12:15.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-08T08:12:15.293 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-08T08:12:15.294 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-08T08:12:15.294 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-08T08:12:15.294 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-08T08:12:15.294 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2023-11-08T08:12:15.294 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-08T08:12:15.295 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2023-11-08T08:12:15.295 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling module streams: 2023-11-08T08:12:15.295 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql 10 2023-11-08T08:12:15.295 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:12:15.295 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2023-11-08T08:12:15.296 INFO:teuthology.orchestra.run.smithi129.stdout:==================================================================================================== 2023-11-08T08:12:15.296 INFO:teuthology.orchestra.run.smithi129.stdout:Install 128 Packages 2023-11-08T08:12:15.296 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 19 Packages 2023-11-08T08:12:15.296 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:12:15.297 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 210 M 2023-11-08T08:12:15.297 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2023-11-08T08:12:16.094 INFO:teuthology.orchestra.run.smithi129.stdout:(1/147): ceph-18.0.0-2698.gb820560f.el8.x86_64. 30 kB/s | 6.4 kB 00:00 2023-11-08T08:12:16.569 INFO:teuthology.orchestra.run.smithi129.stdout:(2/147): ceph-base-18.0.0-2698.gb820560f.el8.x8 7.8 MB/s | 5.4 MB 00:00 2023-11-08T08:12:16.619 INFO:teuthology.orchestra.run.smithi129.stdout:(3/147): ceph-immutable-object-cache-18.0.0-269 3.2 MB/s | 166 kB 00:00 2023-11-08T08:12:16.653 INFO:teuthology.orchestra.run.smithi129.stdout:(4/147): ceph-fuse-18.0.0-2698.gb820560f.el8.x8 1.5 MB/s | 845 kB 00:00 2023-11-08T08:12:16.803 INFO:teuthology.orchestra.run.smithi129.stdout:(5/147): ceph-mds-18.0.0-2698.gb820560f.el8.x86 12 MB/s | 2.3 MB 00:00 2023-11-08T08:12:17.128 INFO:teuthology.orchestra.run.smithi129.stdout:(6/147): ceph-mon-18.0.0-2698.gb820560f.el8.x86 14 MB/s | 4.6 MB 00:00 2023-11-08T08:12:17.229 INFO:teuthology.orchestra.run.smithi129.stdout:(7/147): ceph-mgr-18.0.0-2698.gb820560f.el8.x86 2.7 MB/s | 1.5 MB 00:00 2023-11-08T08:12:18.254 INFO:teuthology.orchestra.run.smithi129.stdout:(8/147): ceph-osd-18.0.0-2698.gb820560f.el8.x86 16 MB/s | 18 MB 00:01 2023-11-08T08:12:18.338 INFO:teuthology.orchestra.run.smithi129.stdout:(9/147): ceph-selinux-18.0.0-2698.gb820560f.el8 292 kB/s | 24 kB 00:00 2023-11-08T08:12:19.230 INFO:teuthology.orchestra.run.smithi129.stdout:(10/147): ceph-common-18.0.0-2698.gb820560f.el8 6.4 MB/s | 22 MB 00:03 2023-11-08T08:12:19.305 INFO:teuthology.orchestra.run.smithi129.stdout:(11/147): cephfs-mirror-18.0.0-2698.gb820560f.e 3.4 MB/s | 258 kB 00:00 2023-11-08T08:12:19.405 INFO:teuthology.orchestra.run.smithi129.stdout:(12/147): libcephfs-devel-18.0.0-2698.gb820560f 305 kB/s | 30 kB 00:00 2023-11-08T08:12:19.606 INFO:teuthology.orchestra.run.smithi129.stdout:(13/147): libcephfs2-18.0.0-2698.gb820560f.el8. 3.7 MB/s | 767 kB 00:00 2023-11-08T08:12:19.697 INFO:teuthology.orchestra.run.smithi129.stdout:(14/147): libcephsqlite-18.0.0-2698.gb820560f.e 1.5 MB/s | 137 kB 00:00 2023-11-08T08:12:19.831 INFO:teuthology.orchestra.run.smithi129.stdout:(15/147): ceph-radosgw-18.0.0-2698.gb820560f.el 5.0 MB/s | 13 MB 00:02 2023-11-08T08:12:19.873 INFO:teuthology.orchestra.run.smithi129.stdout:(16/147): librados-devel-18.0.0-2698.gb820560f. 792 kB/s | 139 kB 00:00 2023-11-08T08:12:19.915 INFO:teuthology.orchestra.run.smithi129.stdout:(17/147): libradosstriper1-18.0.0-2698.gb820560 5.6 MB/s | 474 kB 00:00 2023-11-08T08:12:20.007 INFO:teuthology.orchestra.run.smithi129.stdout:(18/147): python3-ceph-argparse-18.0.0-2698.gb8 486 kB/s | 45 kB 00:00 2023-11-08T08:12:20.099 INFO:teuthology.orchestra.run.smithi129.stdout:(19/147): python3-ceph-common-18.0.0-2698.gb820 1.2 MB/s | 117 kB 00:00 2023-11-08T08:12:20.191 INFO:teuthology.orchestra.run.smithi129.stdout:(20/147): python3-cephfs-18.0.0-2698.gb820560f. 2.0 MB/s | 190 kB 00:00 2023-11-08T08:12:20.300 INFO:teuthology.orchestra.run.smithi129.stdout:(21/147): python3-rados-18.0.0-2698.gb820560f.e 3.3 MB/s | 371 kB 00:00 2023-11-08T08:12:20.400 INFO:teuthology.orchestra.run.smithi129.stdout:(22/147): python3-rbd-18.0.0-2698.gb820560f.el8 3.6 MB/s | 364 kB 00:00 2023-11-08T08:12:20.500 INFO:teuthology.orchestra.run.smithi129.stdout:(23/147): python3-rgw-18.0.0-2698.gb820560f.el8 1.0 MB/s | 107 kB 00:00 2023-11-08T08:12:20.584 INFO:teuthology.orchestra.run.smithi129.stdout:(24/147): librgw2-18.0.0-2698.gb820560f.el8.x86 9.1 MB/s | 6.5 MB 00:00 2023-11-08T08:12:20.609 INFO:teuthology.orchestra.run.smithi129.stdout:(25/147): rbd-fuse-18.0.0-2698.gb820560f.el8.x8 870 kB/s | 94 kB 00:00 2023-11-08T08:12:20.710 INFO:teuthology.orchestra.run.smithi129.stdout:(26/147): rbd-nbd-18.0.0-2698.gb820560f.el8.x86 1.7 MB/s | 179 kB 00:00 2023-11-08T08:12:20.801 INFO:teuthology.orchestra.run.smithi129.stdout:(27/147): ceph-grafana-dashboards-18.0.0-2698.g 252 kB/s | 23 kB 00:00 2023-11-08T08:12:20.894 INFO:teuthology.orchestra.run.smithi129.stdout:(28/147): ceph-mgr-cephadm-18.0.0-2698.gb820560 1.3 MB/s | 123 kB 00:00 2023-11-08T08:12:20.977 INFO:teuthology.orchestra.run.smithi129.stdout:(29/147): rbd-mirror-18.0.0-2698.gb820560f.el8. 8.8 MB/s | 3.4 MB 00:00 2023-11-08T08:12:21.211 INFO:teuthology.orchestra.run.smithi129.stdout:(30/147): ceph-mgr-dashboard-18.0.0-2698.gb8205 5.5 MB/s | 1.7 MB 00:00 2023-11-08T08:12:21.611 INFO:teuthology.orchestra.run.smithi129.stdout:(31/147): ceph-test-18.0.0-2698.gb820560f.el8.x 14 MB/s | 47 MB 00:03 2023-11-08T08:12:21.636 INFO:teuthology.orchestra.run.smithi129.stdout:(32/147): ceph-mgr-modules-core-18.0.0-2698.gb8 555 kB/s | 236 kB 00:00 2023-11-08T08:12:21.661 INFO:teuthology.orchestra.run.smithi129.stdout:(33/147): ceph-mgr-rook-18.0.0-2698.gb820560f.e 951 kB/s | 47 kB 00:00 2023-11-08T08:12:21.704 INFO:teuthology.orchestra.run.smithi129.stdout:(34/147): ceph-prometheus-alerts-18.0.0-2698.gb 216 kB/s | 15 kB 00:00 2023-11-08T08:12:21.729 INFO:teuthology.orchestra.run.smithi129.stdout:(35/147): ceph-volume-18.0.0-2698.gb820560f.el8 3.8 MB/s | 263 kB 00:00 2023-11-08T08:12:21.871 INFO:teuthology.orchestra.run.smithi129.stdout:(36/147): cephadm-18.0.0-2698.gb820560f.el8.noa 495 kB/s | 83 kB 00:00 2023-11-08T08:12:21.905 INFO:teuthology.orchestra.run.smithi129.stdout:(37/147): cephfs-top-18.0.0-2698.gb820560f.el8. 103 kB/s | 18 kB 00:00 2023-11-08T08:12:21.988 INFO:teuthology.orchestra.run.smithi129.stdout:(38/147): ceph-mgr-diskprediction-local-18.0.0- 7.3 MB/s | 7.4 MB 00:01 2023-11-08T08:12:22.080 INFO:teuthology.orchestra.run.smithi129.stdout:(39/147): flex-2.6.1-9.el8.x86_64.rpm 3.4 MB/s | 320 kB 00:00 2023-11-08T08:12:22.114 INFO:teuthology.orchestra.run.smithi129.stdout:(40/147): bison-3.0.4-10.el8.x86_64.rpm 3.2 MB/s | 688 kB 00:00 2023-11-08T08:12:22.139 INFO:teuthology.orchestra.run.smithi129.stdout:(41/147): automake-1.16.1-8.el8.noarch.rpm 2.7 MB/s | 740 kB 00:00 2023-11-08T08:12:22.165 INFO:teuthology.orchestra.run.smithi129.stdout:(42/147): jq-1.6-8.el8.x86_64.rpm 2.4 MB/s | 203 kB 00:00 2023-11-08T08:12:22.190 INFO:teuthology.orchestra.run.smithi129.stdout:(43/147): libpq-13.5-1.el8.x86_64.rpm 2.6 MB/s | 198 kB 00:00 2023-11-08T08:12:22.215 INFO:teuthology.orchestra.run.smithi129.stdout:(44/147): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2023-11-08T08:12:22.240 INFO:teuthology.orchestra.run.smithi129.stdout:(45/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-11-08T08:12:22.274 INFO:teuthology.orchestra.run.smithi129.stdout:(46/147): libtool-2.4.6-25.el8.x86_64.rpm 6.3 MB/s | 709 kB 00:00 2023-11-08T08:12:22.299 INFO:teuthology.orchestra.run.smithi129.stdout:(47/147): oniguruma-6.8.2-2.el8.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2023-11-08T08:12:22.533 INFO:teuthology.orchestra.run.smithi129.stdout:(48/147): postgresql-10.23-2.module_el8+572+929 6.7 MB/s | 1.6 MB 00:00 2023-11-08T08:12:22.775 INFO:teuthology.orchestra.run.smithi129.stdout:(49/147): openblas-0.3.15-4.el8.x86_64.rpm 8.8 MB/s | 4.7 MB 00:00 2023-11-08T08:12:22.842 INFO:teuthology.orchestra.run.smithi129.stdout:(50/147): openblas-threads-0.3.15-4.el8.x86_64. 8.6 MB/s | 4.9 MB 00:00 2023-11-08T08:12:22.984 INFO:teuthology.orchestra.run.smithi129.stdout:(51/147): postgresql-contrib-10.23-2.module_el8 2.0 MB/s | 901 kB 00:00 2023-11-08T08:12:23.076 INFO:teuthology.orchestra.run.smithi129.stdout:(52/147): python3-jinja2-2.10.1-3.el8.noarch.rp 5.7 MB/s | 538 kB 00:00 2023-11-08T08:12:23.110 INFO:teuthology.orchestra.run.smithi129.stdout:(53/147): python3-jsonpatch-1.21-2.el8.noarch.r 798 kB/s | 27 kB 00:00 2023-11-08T08:12:23.143 INFO:teuthology.orchestra.run.smithi129.stdout:(54/147): python3-jsonpointer-1.10-11.el8.noarc 602 kB/s | 20 kB 00:00 2023-11-08T08:12:23.193 INFO:teuthology.orchestra.run.smithi129.stdout:(55/147): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-11-08T08:12:23.227 INFO:teuthology.orchestra.run.smithi129.stdout:(56/147): python3-markupsafe-0.23-19.el8.x86_64 1.1 MB/s | 39 kB 00:00 2023-11-08T08:12:23.326 INFO:teuthology.orchestra.run.smithi129.stdout:(57/147): python3-babel-2.5.1-7.el8.noarch.rpm 9.9 MB/s | 4.8 MB 00:00 2023-11-08T08:12:23.369 INFO:teuthology.orchestra.run.smithi129.stdout:(58/147): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.1 MB/s | 225 kB 00:00 2023-11-08T08:12:23.403 INFO:teuthology.orchestra.run.smithi129.stdout:(59/147): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2023-11-08T08:12:23.454 INFO:teuthology.orchestra.run.smithi129.stdout:(60/147): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-11-08T08:12:23.487 INFO:teuthology.orchestra.run.smithi129.stdout:(61/147): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.7 MB/s | 126 kB 00:00 2023-11-08T08:12:23.529 INFO:teuthology.orchestra.run.smithi129.stdout:(62/147): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-11-08T08:12:23.563 INFO:teuthology.orchestra.run.smithi129.stdout:(63/147): python3-pytz-2017.2-11.el8.noarch.rpm 1.7 MB/s | 57 kB 00:00 2023-11-08T08:12:23.688 INFO:teuthology.orchestra.run.smithi129.stdout:(64/147): postgresql-server-10.23-2.module_el8+ 5.6 MB/s | 5.1 MB 00:00 2023-11-08T08:12:23.747 INFO:teuthology.orchestra.run.smithi129.stdout:(65/147): python3-werkzeug-0.12.2-4.el8.noarch. 7.7 MB/s | 457 kB 00:00 2023-11-08T08:12:23.797 INFO:teuthology.orchestra.run.smithi129.stdout:(66/147): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-11-08T08:12:23.881 INFO:teuthology.orchestra.run.smithi129.stdout:(67/147): python3-numpy-1.14.3-10.el8.x86_64.rp 5.7 MB/s | 3.7 MB 00:00 2023-11-08T08:12:23.906 INFO:teuthology.orchestra.run.smithi129.stdout:(68/147): uuid-1.6.2-43.el8.x86_64.rpm 588 kB/s | 64 kB 00:00 2023-11-08T08:12:23.931 INFO:teuthology.orchestra.run.smithi129.stdout:(69/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2023-11-08T08:12:24.115 INFO:teuthology.orchestra.run.smithi129.stdout:(70/147): elfutils-libelf-devel-0.189-3.el8.x86 296 kB/s | 62 kB 00:00 2023-11-08T08:12:24.140 INFO:teuthology.orchestra.run.smithi129.stdout:(71/147): fuse-2.9.7-17.el8.x86_64.rpm 398 kB/s | 83 kB 00:00 2023-11-08T08:12:24.182 INFO:teuthology.orchestra.run.smithi129.stdout:(72/147): keyutils-libs-devel-1.5.10-9.el8.x86_ 718 kB/s | 48 kB 00:00 2023-11-08T08:12:24.241 INFO:teuthology.orchestra.run.smithi129.stdout:(73/147): libaio-devel-0.3.112-1.el8.x86_64.rpm 320 kB/s | 19 kB 00:00 2023-11-08T08:12:24.300 INFO:teuthology.orchestra.run.smithi129.stdout:(74/147): libcom_err-devel-1.45.6-5.el8.x86_64. 665 kB/s | 39 kB 00:00 2023-11-08T08:12:24.325 INFO:teuthology.orchestra.run.smithi129.stdout:(75/147): krb5-devel-1.18.2-22.el8.x86_64.rpm 3.0 MB/s | 561 kB 00:00 2023-11-08T08:12:24.392 INFO:teuthology.orchestra.run.smithi129.stdout:(76/147): libquadmath-8.5.0-21.el8.x86_64.rpm 2.5 MB/s | 172 kB 00:00 2023-11-08T08:12:24.559 INFO:teuthology.orchestra.run.smithi129.stdout:(77/147): python3-scipy-1.0.0-21.module_el8.5.0 14 MB/s | 14 MB 00:00 2023-11-08T08:12:24.584 INFO:teuthology.orchestra.run.smithi129.stdout:(78/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 244 kB/s | 47 kB 00:00 2023-11-08T08:12:24.618 INFO:teuthology.orchestra.run.smithi129.stdout:(79/147): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2023-11-08T08:12:24.643 INFO:teuthology.orchestra.run.smithi129.stdout:(80/147): libsepol-devel-2.9-3.el8.x86_64.rpm 1.4 MB/s | 87 kB 00:00 2023-11-08T08:12:24.677 INFO:teuthology.orchestra.run.smithi129.stdout:(81/147): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.7 MB/s | 100 kB 00:00 2023-11-08T08:12:24.723 INFO:teuthology.orchestra.run.smithi129.stdout:(82/147): libverto-devel-0.3.2-2.el8.x86_64.rpm 228 kB/s | 18 kB 00:00 2023-11-08T08:12:24.744 INFO:teuthology.orchestra.run.smithi129.stdout:(83/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 655 kB/s | 44 kB 00:00 2023-11-08T08:12:24.828 INFO:teuthology.orchestra.run.smithi129.stdout:(84/147): libselinux-devel-2.9-8.el8.x86_64.rpm 747 kB/s | 200 kB 00:00 2023-11-08T08:12:24.853 INFO:teuthology.orchestra.run.smithi129.stdout:(85/147): pcre2-devel-10.32-3.el8.x86_64.rpm 5.4 MB/s | 605 kB 00:00 2023-11-08T08:12:24.920 INFO:teuthology.orchestra.run.smithi129.stdout:(86/147): pcre2-utf32-10.32-3.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2023-11-08T08:12:24.945 INFO:teuthology.orchestra.run.smithi129.stdout:(87/147): pcre2-utf16-10.32-3.el8.x86_64.rpm 1.9 MB/s | 229 kB 00:00 2023-11-08T08:12:24.995 INFO:teuthology.orchestra.run.smithi129.stdout:(88/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 3.2 MB/s | 249 kB 00:00 2023-11-08T08:12:25.054 INFO:teuthology.orchestra.run.smithi129.stdout:(89/147): openssl-devel-1.1.1k-9.el8.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2023-11-08T08:12:25.080 INFO:teuthology.orchestra.run.smithi129.stdout:(90/147): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2023-11-08T08:12:25.105 INFO:teuthology.orchestra.run.smithi129.stdout:(91/147): python3-cryptography-3.2.1-6.el8.x86_ 3.9 MB/s | 641 kB 00:00 2023-11-08T08:12:25.130 INFO:teuthology.orchestra.run.smithi129.stdout:(92/147): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2023-11-08T08:12:25.155 INFO:teuthology.orchestra.run.smithi129.stdout:(93/147): python3-pycparser-2.14-14.el8.noarch. 1.4 MB/s | 109 kB 00:00 2023-11-08T08:12:25.180 INFO:teuthology.orchestra.run.smithi129.stdout:(94/147): python3-requests-oauthlib-1.0.0-1.el8 565 kB/s | 43 kB 00:00 2023-11-08T08:12:25.206 INFO:teuthology.orchestra.run.smithi129.stdout:(95/147): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 988 kB/s | 74 kB 00:00 2023-11-08T08:12:25.231 INFO:teuthology.orchestra.run.smithi129.stdout:(96/147): zlib-devel-1.2.11-21.el8.x86_64.rpm 769 kB/s | 58 kB 00:00 2023-11-08T08:12:25.306 INFO:teuthology.orchestra.run.smithi129.stdout:(97/147): liboath-2.6.2-3.el8.x86_64.rpm 587 kB/s | 59 kB 00:00 2023-11-08T08:12:25.332 INFO:teuthology.orchestra.run.smithi129.stdout:(98/147): libunwind-1.3.1-3.el8.x86_64.rpm 748 kB/s | 75 kB 00:00 2023-11-08T08:12:25.365 INFO:teuthology.orchestra.run.smithi129.stdout:(99/147): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-11-08T08:12:25.391 INFO:teuthology.orchestra.run.smithi129.stdout:(100/147): python3-bcrypt-3.1.6-2.el8.1.x86_64. 745 kB/s | 44 kB 00:00 2023-11-08T08:12:25.433 INFO:teuthology.orchestra.run.smithi129.stdout:(101/147): python3-beautifulsoup4-4.6.3-2.el8.1 2.7 MB/s | 185 kB 00:00 2023-11-08T08:12:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:(102/147): python3-cachetools-3.1.1-4.el8.noarc 494 kB/s | 33 kB 00:00 2023-11-08T08:12:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:(103/147): python3-certifi-2018.10.15-7.el8.noa 324 kB/s | 16 kB 00:00 2023-11-08T08:12:25.509 INFO:teuthology.orchestra.run.smithi129.stdout:(104/147): python3-asyncssh-2.7.0-2.el8.noarch. 2.3 MB/s | 476 kB 00:00 2023-11-08T08:12:25.534 INFO:teuthology.orchestra.run.smithi129.stdout:(105/147): python3-cheroot-8.5.2-1.el8.noarch.r 2.2 MB/s | 173 kB 00:00 2023-11-08T08:12:25.559 INFO:teuthology.orchestra.run.smithi129.stdout:(106/147): python3-google-auth-1.1.1-10.el8.noa 1.6 MB/s | 81 kB 00:00 2023-11-08T08:12:25.585 INFO:teuthology.orchestra.run.smithi129.stdout:(107/147): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-11-08T08:12:25.610 INFO:teuthology.orchestra.run.smithi129.stdout:(108/147): python3-jaraco-functools-2.0-4.el8.n 350 kB/s | 18 kB 00:00 2023-11-08T08:12:25.635 INFO:teuthology.orchestra.run.smithi129.stdout:(109/147): python3-cherrypy-18.4.0-1.el8.noarch 2.5 MB/s | 384 kB 00:00 2023-11-08T08:12:25.660 INFO:teuthology.orchestra.run.smithi129.stdout:(110/147): python3-logutils-0.3.5-11.el8.noarch 971 kB/s | 49 kB 00:00 2023-11-08T08:12:25.686 INFO:teuthology.orchestra.run.smithi129.stdout:(111/147): python3-more-itertools-7.2.0-3.el8.n 1.1 MB/s | 59 kB 00:00 2023-11-08T08:12:25.711 INFO:teuthology.orchestra.run.smithi129.stdout:(112/147): python3-natsort-7.1.1-2.el8.noarch.r 1.2 MB/s | 60 kB 00:00 2023-11-08T08:12:25.736 INFO:teuthology.orchestra.run.smithi129.stdout:(113/147): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-11-08T08:12:25.762 INFO:teuthology.orchestra.run.smithi129.stdout:(114/147): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-11-08T08:12:25.787 INFO:teuthology.orchestra.run.smithi129.stdout:(115/147): python3-repoze-lru-0.7-6.el8.noarch. 667 kB/s | 34 kB 00:00 2023-11-08T08:12:25.812 INFO:teuthology.orchestra.run.smithi129.stdout:(116/147): python3-routes-2.4.1-12.el8.noarch.r 3.8 MB/s | 196 kB 00:00 2023-11-08T08:12:25.838 INFO:teuthology.orchestra.run.smithi129.stdout:(117/147): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-11-08T08:12:25.854 INFO:teuthology.orchestra.run.smithi129.stdout:(118/147): python3-simplegeneric-0.8.1-17.el8.n 447 kB/s | 19 kB 00:00 2023-11-08T08:12:25.871 INFO:teuthology.orchestra.run.smithi129.stdout:(119/147): python3-singledispatch-3.4.0.3-18.el 715 kB/s | 24 kB 00:00 2023-11-08T08:12:25.888 INFO:teuthology.orchestra.run.smithi129.stdout:(120/147): python3-tempora-1.14.1-5.el8.noarch. 870 kB/s | 29 kB 00:00 2023-11-08T08:12:25.905 INFO:teuthology.orchestra.run.smithi129.stdout:(121/147): python3-trustme-0.6.0-4.el8.noarch.r 819 kB/s | 27 kB 00:00 2023-11-08T08:12:25.939 INFO:teuthology.orchestra.run.smithi129.stdout:(122/147): python3-kubernetes-11.0.0-6.el8.noar 3.3 MB/s | 1.2 MB 00:00 2023-11-08T08:12:25.964 INFO:teuthology.orchestra.run.smithi129.stdout:(123/147): python3-waitress-1.4.3-1.el8.noarch. 3.3 MB/s | 255 kB 00:00 2023-11-08T08:12:25.990 INFO:teuthology.orchestra.run.smithi129.stdout:(124/147): python3-webob-1.8.5-1.el8.1.noarch.r 2.9 MB/s | 251 kB 00:00 2023-11-08T08:12:26.015 INFO:teuthology.orchestra.run.smithi129.stdout:(125/147): python3-websocket-client-0.56.0-5.el 813 kB/s | 61 kB 00:00 2023-11-08T08:12:26.040 INFO:teuthology.orchestra.run.smithi129.stdout:(126/147): python3-zc-lockfile-2.0-2.el8.noarch 458 kB/s | 23 kB 00:00 2023-11-08T08:12:26.065 INFO:teuthology.orchestra.run.smithi129.stdout:(127/147): python3-webtest-2.0.33-1.el8.noarch. 1.1 MB/s | 85 kB 00:00 2023-11-08T08:12:26.157 INFO:teuthology.orchestra.run.smithi129.stdout:(128/147): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2023-11-08T08:12:26.291 INFO:teuthology.orchestra.run.smithi129.stdout:(129/147): NetworkManager-initscripts-updown-1. 1.1 MB/s | 144 kB 00:00 2023-11-08T08:12:26.358 INFO:teuthology.orchestra.run.smithi129.stdout:(130/147): acl-2.2.53-3.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-11-08T08:12:26.408 INFO:teuthology.orchestra.run.smithi129.stdout:(131/147): librbd1-18.0.0-2698.gb820560f.el8.x8 10 MB/s | 3.6 MB 00:00 2023-11-08T08:12:26.434 INFO:teuthology.orchestra.run.smithi129.stdout:(132/147): elfutils-debuginfod-client-0.189-3.e 1.0 MB/s | 76 kB 00:00 2023-11-08T08:12:26.534 INFO:teuthology.orchestra.run.smithi129.stdout:(133/147): elfutils-libs-0.189-3.el8.x86_64.rpm 3.0 MB/s | 303 kB 00:00 2023-11-08T08:12:26.626 INFO:teuthology.orchestra.run.smithi129.stdout:(134/147): librados2-18.0.0-2698.gb820560f.el8. 6.2 MB/s | 3.7 MB 00:00 2023-11-08T08:12:26.651 INFO:teuthology.orchestra.run.smithi129.stdout:(135/147): elfutils-libelf-0.189-3.el8.x86_64.r 957 kB/s | 232 kB 00:00 2023-11-08T08:12:26.693 INFO:teuthology.orchestra.run.smithi129.stdout:(136/147): iproute-6.2.0-5.el8.x86_64.rpm 5.3 MB/s | 854 kB 00:00 2023-11-08T08:12:26.718 INFO:teuthology.orchestra.run.smithi129.stdout:(137/147): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.2 MB/s | 85 kB 00:00 2023-11-08T08:12:26.744 INFO:teuthology.orchestra.run.smithi129.stdout:(138/147): libacl-2.2.53-3.el8.x86_64.rpm 296 kB/s | 35 kB 00:00 2023-11-08T08:12:26.769 INFO:teuthology.orchestra.run.smithi129.stdout:(139/147): libblkid-2.32.1-43.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2023-11-08T08:12:26.853 INFO:teuthology.orchestra.run.smithi129.stdout:(140/147): libfdisk-2.32.1-43.el8.x86_64.rpm 1.9 MB/s | 254 kB 00:00 2023-11-08T08:12:26.878 INFO:teuthology.orchestra.run.smithi129.stdout:(141/147): libmount-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 236 kB 00:00 2023-11-08T08:12:26.903 INFO:teuthology.orchestra.run.smithi129.stdout:(142/147): libsmartcols-2.32.1-43.el8.x86_64.rp 1.3 MB/s | 179 kB 00:00 2023-11-08T08:12:26.928 INFO:teuthology.orchestra.run.smithi129.stdout:(143/147): libuuid-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 99 kB 00:00 2023-11-08T08:12:26.979 INFO:teuthology.orchestra.run.smithi129.stdout:(144/147): util-linux-user-2.32.1-43.el8.x86_64 1.3 MB/s | 103 kB 00:00 2023-11-08T08:12:27.004 INFO:teuthology.orchestra.run.smithi129.stdout:(145/147): xfsdump-3.1.8-7.el8.x86_64.rpm 4.3 MB/s | 334 kB 00:00 2023-11-08T08:12:27.121 INFO:teuthology.orchestra.run.smithi129.stdout:(146/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2023-11-08T08:12:27.172 INFO:teuthology.orchestra.run.smithi129.stdout:(147/147): util-linux-2.32.1-43.el8.x86_64.rpm 9.3 MB/s | 2.5 MB 00:00 2023-11-08T08:12:27.174 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2023-11-08T08:12:27.175 INFO:teuthology.orchestra.run.smithi129.stdout:Total 18 MB/s | 210 MB 00:11 2023-11-08T08:12:28.119 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2023-11-08T08:12:28.562 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2023-11-08T08:12:28.562 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2023-11-08T08:12:29.987 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2023-11-08T08:12:29.991 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2023-11-08T08:12:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2023-11-08T08:12:35.680 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: thrift-0.13.0-2.el8.x86_64 1/1 2023-11-08T08:12:35.869 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/166 2023-11-08T08:12:36.053 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 2/166 2023-11-08T08:12:36.115 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 2/166 2023-11-08T08:12:36.287 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 3/166 2023-11-08T08:12:36.351 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 3/166 2023-11-08T08:12:36.570 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/166 2023-11-08T08:12:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 5/166 2023-11-08T08:12:37.007 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/166 2023-11-08T08:12:37.270 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/166 2023-11-08T08:12:37.678 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/166 2023-11-08T08:12:37.869 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:18.0.0-2698.gb820560f.el8.x86_64 8/166 2023-11-08T08:12:38.245 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:18.0.0-2698.gb820560f.el8.x86_64 8/166 2023-11-08T08:12:38.444 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 9/166 2023-11-08T08:12:38.565 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 9/166 2023-11-08T08:12:38.753 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 10/166 2023-11-08T08:12:38.880 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 10/166 2023-11-08T08:12:39.092 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:18.0.0-2698.gb820560f.el8.x86_64 11/166 2023-11-08T08:12:39.261 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 12/166 2023-11-08T08:12:39.455 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 12/166 2023-11-08T08:12:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/166 2023-11-08T08:12:39.819 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/166 2023-11-08T08:12:40.033 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 14/166 2023-11-08T08:12:40.234 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 15/166 2023-11-08T08:12:40.657 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpq-13.5-1.el8.x86_64 16/166 2023-11-08T08:12:40.847 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 17/166 2023-11-08T08:12:41.019 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 18/166 2023-11-08T08:12:41.122 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 18/166 2023-11-08T08:12:41.252 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86 19/166 2023-11-08T08:12:41.317 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86 19/166 2023-11-08T08:12:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 20/166 2023-11-08T08:12:41.775 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 21/166 2023-11-08T08:12:41.957 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 22/166 2023-11-08T08:12:42.119 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 23/166 2023-11-08T08:12:42.260 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 23/166 2023-11-08T08:12:42.627 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 24/166 2023-11-08T08:12:43.238 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 24/166 2023-11-08T08:12:43.452 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.15-4.el8.x86_64 25/166 2023-11-08T08:12:44.063 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 25/166 2023-11-08T08:12:44.202 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 26/166 2023-11-08T08:12:44.893 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 26/166 2023-11-08T08:12:45.211 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 27/166 2023-11-08T08:12:45.416 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 28/166 2023-11-08T08:12:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 29/166 2023-11-08T08:12:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 30/166 2023-11-08T08:12:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:18.0.0-2698.gb820560f.el8. 31/166 2023-11-08T08:12:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:18.0.0-2698.gb820560f.el 32/166 2023-11-08T08:12:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:18.0.0-2698.gb820560f.el8.x86_6 33/166 2023-11-08T08:12:47.439 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/166 2023-11-08T08:12:47.785 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/166 2023-11-08T08:12:48.065 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/166 2023-11-08T08:12:48.268 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/166 2023-11-08T08:12:50.048 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/166 2023-11-08T08:12:50.364 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 38/166 2023-11-08T08:12:50.552 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 39/166 2023-11-08T08:12:50.778 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/166 2023-11-08T08:12:50.999 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:18.0.0-2698.gb820560f.el8.x86_64 41/166 2023-11-08T08:12:51.200 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:18.0.0-2698.gb820560f.el8.x86_6 42/166 2023-11-08T08:12:51.364 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 43/166 2023-11-08T08:12:51.571 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-debuginfod-client-0.189-3.el8.x86_64 44/166 2023-11-08T08:12:51.809 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 45/166 2023-11-08T08:12:51.885 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 45/166 2023-11-08T08:12:52.068 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 46/166 2023-11-08T08:12:52.137 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 46/166 2023-11-08T08:12:52.291 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/166 2023-11-08T08:12:52.484 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 48/166 2023-11-08T08:12:53.110 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 48/166 2023-11-08T08:12:53.560 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 49/166 2023-11-08T08:12:53.575 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 49/166 2023-11-08T08:12:53.796 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 50/166 2023-11-08T08:12:54.062 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 51/166 2023-11-08T08:12:54.290 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 52/166 2023-11-08T08:12:54.505 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 53/166 2023-11-08T08:12:54.670 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 54/166 2023-11-08T08:12:54.853 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 55/166 2023-11-08T08:12:55.072 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 56/166 2023-11-08T08:12:55.313 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 57/166 2023-11-08T08:12:55.470 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 58/166 2023-11-08T08:12:55.623 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 59/166 2023-11-08T08:12:55.788 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 60/166 2023-11-08T08:12:55.965 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 61/166 2023-11-08T08:12:56.170 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-2.6-1.el8.noarch 62/166 2023-11-08T08:12:56.334 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 63/166 2023-11-08T08:12:56.535 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 64/166 2023-11-08T08:12:56.775 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 65/166 2023-11-08T08:12:56.989 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 66/166 2023-11-08T08:12:57.363 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 67/166 2023-11-08T08:12:57.609 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 68/166 2023-11-08T08:12:57.792 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 69/166 2023-11-08T08:12:58.559 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 70/166 2023-11-08T08:12:58.783 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 71/166 2023-11-08T08:12:58.858 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 71/166 2023-11-08T08:12:59.062 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:18.0.0-2698.gb820560f.el8.x86_64 72/166 2023-11-08T08:12:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 73/166 2023-11-08T08:12:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 74/166 2023-11-08T08:12:59.987 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 75/166 2023-11-08T08:13:00.219 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 76/166 2023-11-08T08:13:00.474 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 77/166 2023-11-08T08:13:00.721 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 78/166 2023-11-08T08:13:01.800 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 79/166 2023-11-08T08:13:02.150 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 80/166 2023-11-08T08:13:02.391 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 81/166 2023-11-08T08:13:02.617 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 82/166 2023-11-08T08:13:02.948 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 83/166 2023-11-08T08:13:03.184 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 84/166 2023-11-08T08:13:03.473 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 85/166 2023-11-08T08:13:03.727 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 86/166 2023-11-08T08:13:04.036 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 87/166 2023-11-08T08:13:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 88/166 2023-11-08T08:13:04.451 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 89/166 2023-11-08T08:13:04.647 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 90/166 2023-11-08T08:13:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 91/166 2023-11-08T08:13:05.186 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 92/166 2023-11-08T08:13:05.338 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 93/166 2023-11-08T08:13:05.571 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 94/166 2023-11-08T08:13:05.810 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 95/166 2023-11-08T08:13:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.7-17.el8.x86_64 96/166 2023-11-08T08:13:06.189 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 97/166 2023-11-08T08:13:06.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-43.el8.x86_64 98/166 2023-11-08T08:13:06.453 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 98/166 2023-11-08T08:13:06.642 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 99/166 2023-11-08T08:13:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 100/166 2023-11-08T08:13:09.402 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 101/166 2023-11-08T08:13:09.624 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 101/166 2023-11-08T08:13:10.198 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 101/166 2023-11-08T08:13:10.538 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 102/166 2023-11-08T08:13:10.641 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 102/166 2023-11-08T08:13:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 103/166 2023-11-08T08:13:29.604 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 103/166 2023-11-08T08:13:29.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2023-11-08T08:13:29.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2023-11-08T08:13:29.605 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /dev 2023-11-08T08:13:29.606 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /run 2023-11-08T08:13:29.606 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2023-11-08T08:13:29.606 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2023-11-08T08:13:29.606 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2023-11-08T08:13:29.606 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2023-11-08T08:13:29.607 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /dev 2023-11-08T08:13:29.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:13:29.946 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 104/166 2023-11-08T08:13:31.509 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 104/166 2023-11-08T08:13:31.726 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 105/166 2023-11-08T08:13:31.943 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 105/166 2023-11-08T08:13:32.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 106/166 2023-11-08T08:13:32.569 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 106/166 2023-11-08T08:13:32.737 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 107/166 2023-11-08T08:13:32.760 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 107/166 2023-11-08T08:13:32.985 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 108/166 2023-11-08T08:13:33.189 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 109/166 2023-11-08T08:13:33.337 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 110/166 2023-11-08T08:13:33.419 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 110/166 2023-11-08T08:13:33.729 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-8.el8.x86_64 111/166 2023-11-08T08:13:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.1-8.el8.noarch 112/166 2023-11-08T08:13:34.467 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:18.0.0-2698.gb820560f.el8.noarch 113/166 2023-11-08T08:13:34.653 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:18.0.0-2698.gb820560f.el8.noarch 113/166 2023-11-08T08:13:34.792 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2698.gb820560f.e 114/166 2023-11-08T08:13:34.967 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2698.gb820560f. 115/166 2023-11-08T08:13:35.195 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noa 116/166 2023-11-08T08:13:35.407 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noa 116/166 2023-11-08T08:13:35.625 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.n 117/166 2023-11-08T08:13:36.584 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.n 117/166 2023-11-08T08:13:36.778 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2698.gb82 118/166 2023-11-08T08:13:36.869 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2698.gb82 118/166 2023-11-08T08:13:37.288 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2698.gb820560f.el 119/166 2023-11-08T08:13:37.517 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 120/166 2023-11-08T08:13:37.552 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 120/166 2023-11-08T08:13:37.736 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 121/166 2023-11-08T08:13:37.747 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 121/166 2023-11-08T08:13:37.993 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:18.0.0-2698.gb820560f.el8.x86_64 122/166 2023-11-08T08:13:38.136 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-25.el8.x86_64 123/166 2023-11-08T08:13:42.511 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 123/166 2023-11-08T08:13:43.950 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:18.0.0-2698.gb820560f.el8.x86_64 124/166 2023-11-08T08:13:44.183 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 125/166 2023-11-08T08:13:44.217 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 125/166 2023-11-08T08:13:44.411 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2698.gb8205 126/166 2023-11-08T08:13:44.452 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2698.gb8205 126/166 2023-11-08T08:13:44.652 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 127/166 2023-11-08T08:13:44.996 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 127/166 2023-11-08T08:13:45.177 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 128/166 2023-11-08T08:13:45.387 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 128/166 2023-11-08T08:13:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 129/166 2023-11-08T08:13:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 130/166 2023-11-08T08:13:46.898 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:1.1.1k-9.el8.x86_64 131/166 2023-11-08T08:13:47.090 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 132/166 2023-11-08T08:13:47.297 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/166 2023-11-08T08:13:47.464 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/166 2023-11-08T08:13:47.690 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:18.0.0-2698.gb820560f.el8.x86_ 135/166 2023-11-08T08:13:47.968 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 136/166 2023-11-08T08:13:48.152 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 137/166 2023-11-08T08:13:49.175 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 138/166 2023-11-08T08:13:49.452 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 138/166 2023-11-08T08:13:49.515 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 138/166 2023-11-08T08:13:49.730 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 139/166 2023-11-08T08:13:49.943 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:18.0.0-2698.gb820560f.el8.noarch 140/166 2023-11-08T08:13:50.184 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 141/166 2023-11-08T08:13:50.533 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:18.0.0-2698.gb820560f.el8.x86_64 142/166 2023-11-08T08:13:50.820 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 143/166 2023-11-08T08:13:50.967 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-11.e 144/166 2023-11-08T08:13:50.988 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-11.e 144/166 2023-11-08T08:13:51.223 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 145/166 2023-11-08T08:13:51.459 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.1-9.el8.x86_64 146/166 2023-11-08T08:13:51.556 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 146/166 2023-11-08T08:13:51.792 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.0.4-10.el8.x86_64 147/166 2023-11-08T08:13:51.806 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 147/166 2023-11-08T08:13:52.028 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 148/166 2023-11-08T08:13:52.192 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 149/166 2023-11-08T08:13:52.383 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/166 2023-11-08T08:13:52.383 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/166 2023-11-08T08:13:52.654 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/166 2023-11-08T08:13:52.655 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/166 2023-11-08T08:13:52.880 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/166 2023-11-08T08:13:52.994 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 153/166 2023-11-08T08:13:53.000 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 154/166 2023-11-08T08:13:53.113 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 154/166 2023-11-08T08:13:53.114 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 155/166 2023-11-08T08:13:53.396 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 155/166 2023-11-08T08:13:53.564 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 156/166 2023-11-08T08:13:53.753 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 157/166 2023-11-08T08:13:53.899 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 158/166 2023-11-08T08:13:54.128 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 159/166 2023-11-08T08:13:54.128 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 160/166 2023-11-08T08:13:54.329 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 160/166 2023-11-08T08:13:54.329 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 161/166 2023-11-08T08:13:54.520 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 161/166 2023-11-08T08:13:54.521 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 162/166 2023-11-08T08:13:54.754 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 162/166 2023-11-08T08:13:54.754 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 163/166 2023-11-08T08:13:54.962 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 163/166 2023-11-08T08:13:54.962 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 164/166 2023-11-08T08:13:55.145 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 164/166 2023-11-08T08:13:55.327 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 165/166 2023-11-08T08:13:55.510 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 166/166 2023-11-08T08:13:56.091 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libacl-2.2.53-1.el8.x86_64 166/166 2023-11-08T08:13:56.091 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:18.0.0-2698.gb820560f.el8.x86_64 1/166 2023-11-08T08:13:56.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 2/166 2023-11-08T08:13:56.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 3/166 2023-11-08T08:13:56.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 4/166 2023-11-08T08:13:56.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2698.gb8205 5/166 2023-11-08T08:13:56.092 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 6/166 2023-11-08T08:13:56.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 7/166 2023-11-08T08:13:56.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 8/166 2023-11-08T08:13:56.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 9/166 2023-11-08T08:13:56.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 10/166 2023-11-08T08:13:56.093 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 11/166 2023-11-08T08:13:56.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:18.0.0-2698.gb820560f.el8.x86_64 12/166 2023-11-08T08:13:56.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 13/166 2023-11-08T08:13:56.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:18.0.0-2698.gb820560f.el8.x86_ 14/166 2023-11-08T08:13:56.094 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 15/166 2023-11-08T08:13:56.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 16/166 2023-11-08T08:13:56.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:18.0.0-2698.gb820560f.el8.x86_6 17/166 2023-11-08T08:13:56.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86 18/166 2023-11-08T08:13:56.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 19/166 2023-11-08T08:13:56.095 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2698.gb820560f.el 20/166 2023-11-08T08:13:56.096 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:18.0.0-2698.gb820560f.el8. 21/166 2023-11-08T08:13:56.096 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:18.0.0-2698.gb820560f.el8.x86_6 22/166 2023-11-08T08:13:56.096 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:18.0.0-2698.gb820560f.el8.x86_64 23/166 2023-11-08T08:13:56.096 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:18.0.0-2698.gb820560f.el8.x86_64 24/166 2023-11-08T08:13:56.096 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:18.0.0-2698.gb820560f.el8.x86_64 25/166 2023-11-08T08:13:56.097 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 26/166 2023-11-08T08:13:56.097 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 27/166 2023-11-08T08:13:56.097 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:18.0.0-2698.gb820560f.el8.x86_64 28/166 2023-11-08T08:13:56.097 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2698.gb820560f. 29/166 2023-11-08T08:13:56.097 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noa 30/166 2023-11-08T08:13:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.n 31/166 2023-11-08T08:13:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2698.gb82 32/166 2023-11-08T08:13:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2698.gb820560f.el 33/166 2023-11-08T08:13:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 34/166 2023-11-08T08:13:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2698.gb820560f.e 35/166 2023-11-08T08:13:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 36/166 2023-11-08T08:13:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:18.0.0-2698.gb820560f.el8.noarch 37/166 2023-11-08T08:13:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:18.0.0-2698.gb820560f.el8.noarch 38/166 2023-11-08T08:13:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/166 2023-11-08T08:13:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/166 2023-11-08T08:13:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/166 2023-11-08T08:13:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-8.el8.x86_64 42/166 2023-11-08T08:13:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/166 2023-11-08T08:13:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 44/166 2023-11-08T08:13:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/166 2023-11-08T08:13:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/166 2023-11-08T08:13:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 47/166 2023-11-08T08:13:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/166 2023-11-08T08:13:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/166 2023-11-08T08:13:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/166 2023-11-08T08:13:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/166 2023-11-08T08:13:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/166 2023-11-08T08:13:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/166 2023-11-08T08:13:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 54/166 2023-11-08T08:13:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/166 2023-11-08T08:13:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/166 2023-11-08T08:13:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/166 2023-11-08T08:13:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/166 2023-11-08T08:13:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/166 2023-11-08T08:13:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/166 2023-11-08T08:13:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/166 2023-11-08T08:13:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/166 2023-11-08T08:13:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/166 2023-11-08T08:13:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/166 2023-11-08T08:13:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/166 2023-11-08T08:13:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 66/166 2023-11-08T08:13:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/166 2023-11-08T08:13:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/166 2023-11-08T08:13:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/166 2023-11-08T08:13:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/166 2023-11-08T08:13:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 71/166 2023-11-08T08:13:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 72/166 2023-11-08T08:13:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/166 2023-11-08T08:13:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/166 2023-11-08T08:13:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/166 2023-11-08T08:13:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/166 2023-11-08T08:13:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/166 2023-11-08T08:13:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/166 2023-11-08T08:13:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 79/166 2023-11-08T08:13:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/166 2023-11-08T08:13:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/166 2023-11-08T08:13:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 82/166 2023-11-08T08:13:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/166 2023-11-08T08:13:56.121 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/166 2023-11-08T08:13:56.121 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:1.1.1k-9.el8.x86_64 85/166 2023-11-08T08:13:56.121 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/166 2023-11-08T08:13:56.121 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/166 2023-11-08T08:13:56.121 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/166 2023-11-08T08:13:56.122 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/166 2023-11-08T08:13:56.122 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 90/166 2023-11-08T08:13:56.122 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/166 2023-11-08T08:13:56.122 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/166 2023-11-08T08:13:56.122 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/166 2023-11-08T08:13:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/166 2023-11-08T08:13:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/166 2023-11-08T08:13:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/166 2023-11-08T08:13:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 97/166 2023-11-08T08:13:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 98/166 2023-11-08T08:13:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 99/166 2023-11-08T08:13:56.124 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 100/166 2023-11-08T08:13:56.124 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 101/166 2023-11-08T08:13:56.124 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 102/166 2023-11-08T08:13:56.124 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 103/166 2023-11-08T08:13:56.124 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 104/166 2023-11-08T08:13:56.125 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 105/166 2023-11-08T08:13:56.125 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 106/166 2023-11-08T08:13:56.125 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 107/166 2023-11-08T08:13:56.125 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 108/166 2023-11-08T08:13:56.125 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 109/166 2023-11-08T08:13:56.126 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 110/166 2023-11-08T08:13:56.126 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 111/166 2023-11-08T08:13:56.126 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 112/166 2023-11-08T08:13:56.126 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 113/166 2023-11-08T08:13:56.126 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 114/166 2023-11-08T08:13:56.127 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-2.6-1.el8.noarch 115/166 2023-11-08T08:13:56.127 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 116/166 2023-11-08T08:13:56.127 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 117/166 2023-11-08T08:13:56.127 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 118/166 2023-11-08T08:13:56.127 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 119/166 2023-11-08T08:13:56.128 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 120/166 2023-11-08T08:13:56.128 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 121/166 2023-11-08T08:13:56.128 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 122/166 2023-11-08T08:13:56.128 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 123/166 2023-11-08T08:13:56.128 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 124/166 2023-11-08T08:13:56.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 125/166 2023-11-08T08:13:56.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 126/166 2023-11-08T08:13:56.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 127/166 2023-11-08T08:13:56.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 128/166 2023-11-08T08:13:56.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:18.0.0-2698.gb820560f.el8.x86_64 129/166 2023-11-08T08:13:56.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/166 2023-11-08T08:13:56.130 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 131/166 2023-11-08T08:13:56.130 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/166 2023-11-08T08:13:56.130 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-11.e 133/166 2023-11-08T08:13:56.130 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 134/166 2023-11-08T08:13:56.131 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : acl-2.2.53-3.el8.x86_64 135/166 2023-11-08T08:13:56.131 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : acl-2.2.53-1.el8.x86_64 136/166 2023-11-08T08:13:56.131 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el8.x86_64 137/166 2023-11-08T08:13:56.131 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 138/166 2023-11-08T08:13:56.131 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 139/166 2023-11-08T08:13:56.131 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 140/166 2023-11-08T08:13:56.132 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 141/166 2023-11-08T08:13:56.132 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 142/166 2023-11-08T08:13:56.132 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 143/166 2023-11-08T08:13:56.132 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 144/166 2023-11-08T08:13:56.132 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 145/166 2023-11-08T08:13:56.133 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 146/166 2023-11-08T08:13:56.133 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 147/166 2023-11-08T08:13:56.133 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 148/166 2023-11-08T08:13:56.133 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 149/166 2023-11-08T08:13:56.133 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 150/166 2023-11-08T08:13:56.133 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 151/166 2023-11-08T08:13:56.134 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 152/166 2023-11-08T08:13:56.134 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 153/166 2023-11-08T08:13:56.134 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 154/166 2023-11-08T08:13:56.134 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 155/166 2023-11-08T08:13:56.134 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 156/166 2023-11-08T08:13:56.135 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 157/166 2023-11-08T08:13:56.135 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 158/166 2023-11-08T08:13:56.135 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 159/166 2023-11-08T08:13:56.135 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 160/166 2023-11-08T08:13:56.135 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 161/166 2023-11-08T08:13:56.135 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 162/166 2023-11-08T08:13:56.136 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 163/166 2023-11-08T08:13:56.136 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 164/166 2023-11-08T08:13:56.136 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 165/166 2023-11-08T08:13:58.410 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 166/166 2023-11-08T08:13:58.410 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:13:58.411 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2023-11-08T08:13:58.411 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-initscripts-updown-1:1.40.16-11.el8.noarch 2023-11-08T08:13:58.411 INFO:teuthology.orchestra.run.smithi129.stdout: acl-2.2.53-3.el8.x86_64 2023-11-08T08:13:58.411 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client-0.189-3.el8.x86_64 2023-11-08T08:13:58.411 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-0.189-3.el8.x86_64 2023-11-08T08:13:58.412 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs-0.189-3.el8.x86_64 2023-11-08T08:13:58.412 INFO:teuthology.orchestra.run.smithi129.stdout: iproute-6.2.0-5.el8.x86_64 2023-11-08T08:13:58.412 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-2.2.53-3.el8.x86_64 2023-11-08T08:13:58.412 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-devel-2.2.53-3.el8.x86_64 2023-11-08T08:13:58.413 INFO:teuthology.orchestra.run.smithi129.stdout: libblkid-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.413 INFO:teuthology.orchestra.run.smithi129.stdout: libfdisk-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.413 INFO:teuthology.orchestra.run.smithi129.stdout: libmount-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.413 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.413 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.414 INFO:teuthology.orchestra.run.smithi129.stdout: libsmartcols-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.414 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.414 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.414 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-user-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.415 INFO:teuthology.orchestra.run.smithi129.stdout: xfsdump-3.1.8-7.el8.x86_64 2023-11-08T08:13:58.415 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-5.0.0-12.el8.x86_64 2023-11-08T08:13:58.415 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2023-11-08T08:13:58.416 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.1-8.el8.noarch 2023-11-08T08:13:58.416 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.0.4-10.el8.x86_64 2023-11-08T08:13:58.416 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.416 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.417 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.417 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.417 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.417 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.418 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.418 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.418 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.418 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.419 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.419 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.419 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.419 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.420 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.420 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.420 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.420 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.420 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.421 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.421 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.421 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.421 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:13:58.422 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2023-11-08T08:13:58.422 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.1-9.el8.x86_64 2023-11-08T08:13:58.422 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.7-17.el8.x86_64 2023-11-08T08:13:58.422 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-08T08:13:58.423 INFO:teuthology.orchestra.run.smithi129.stdout: jq-1.6-8.el8.x86_64 2023-11-08T08:13:58.423 INFO:teuthology.orchestra.run.smithi129.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-08T08:13:58.423 INFO:teuthology.orchestra.run.smithi129.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-11-08T08:13:58.423 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-08T08:13:58.423 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.424 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.424 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.424 INFO:teuthology.orchestra.run.smithi129.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-11-08T08:13:58.424 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-08T08:13:58.424 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-08T08:13:58.424 INFO:teuthology.orchestra.run.smithi129.stdout: libpq-13.5-1.el8.x86_64 2023-11-08T08:13:58.425 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-08T08:13:58.425 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-08T08:13:58.425 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.426 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.426 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-08T08:13:58.426 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.426 INFO:teuthology.orchestra.run.smithi129.stdout: libselinux-devel-2.9-8.el8.x86_64 2023-11-08T08:13:58.427 INFO:teuthology.orchestra.run.smithi129.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-08T08:13:58.427 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-08T08:13:58.427 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-08T08:13:58.427 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2023-11-08T08:13:58.427 INFO:teuthology.orchestra.run.smithi129.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-11-08T08:13:58.428 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2023-11-08T08:13:58.428 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-08T08:13:58.428 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-08T08:13:58.428 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-08T08:13:58.429 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-08T08:13:58.429 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:1.1.1k-9.el8.x86_64 2023-11-08T08:13:58.429 INFO:teuthology.orchestra.run.smithi129.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-11-08T08:13:58.429 INFO:teuthology.orchestra.run.smithi129.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-11-08T08:13:58.429 INFO:teuthology.orchestra.run.smithi129.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-11-08T08:13:58.430 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2023-11-08T08:13:58.430 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2023-11-08T08:13:58.430 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2023-11-08T08:13:58.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-08T08:13:58.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-08T08:13:58.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-08T08:13:58.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-08T08:13:58.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-08T08:13:58.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-08T08:13:58.433 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-08T08:13:58.433 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-08T08:13:58.433 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-08T08:13:58.434 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-08T08:13:58.434 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-08T08:13:58.434 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-08T08:13:58.435 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-08T08:13:58.435 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-08T08:13:58.435 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-08T08:13:58.435 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-08T08:13:58.435 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-08T08:13:58.436 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-08T08:13:58.436 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-08T08:13:58.436 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-08T08:13:58.436 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-08T08:13:58.436 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-08T08:13:58.436 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-08T08:13:58.437 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-08T08:13:58.437 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-08T08:13:58.437 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-08T08:13:58.437 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-08T08:13:58.437 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-2.6-1.el8.noarch 2023-11-08T08:13:58.438 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-08T08:13:58.438 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-08T08:13:58.438 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-08T08:13:58.438 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-08T08:13:58.438 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-08T08:13:58.439 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-08T08:13:58.439 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.439 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.439 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-08T08:13:58.439 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-08T08:13:58.440 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.440 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-08T08:13:58.440 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-08T08:13:58.440 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-08T08:13:58.440 INFO:teuthology.orchestra.run.smithi129.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-08T08:13:58.440 INFO:teuthology.orchestra.run.smithi129.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-08T08:13:58.441 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-08T08:13:58.441 INFO:teuthology.orchestra.run.smithi129.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-08T08:13:58.441 INFO:teuthology.orchestra.run.smithi129.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-08T08:13:58.441 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-08T08:13:58.441 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-08T08:13:58.442 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-08T08:13:58.442 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-08T08:13:58.442 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-08T08:13:58.442 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.442 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.442 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:13:58.443 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-08T08:13:58.443 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-08T08:13:58.443 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-08T08:13:58.443 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2023-11-08T08:13:58.443 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-08T08:13:58.444 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-11-08T08:13:58.444 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:13:58.444 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2023-11-08T08:13:58.713 DEBUG:teuthology.parallel:result is None 2023-11-08T08:15:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - Extras 69 B/s | 18 kB 04:23 2023-11-08T08:15:50.813 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - Extras common packages 29 kB/s | 6.9 kB 00:00 2023-11-08T08:15:51.614 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.6 MB 00:00 2023-11-08T08:15:53.033 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-11-08T08:15:58.033 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 3.3 MB/s | 16 MB 00:04 2023-11-08T08:16:01.921 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 301 kB/s | 24 kB 00:00 2023-11-08T08:16:06.273 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-08T08:16:06.274 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-08T08:16:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2023-11-08T08:16:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2023-11-08T08:16:06.289 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2023-11-08T08:16:06.291 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2023-11-08T08:16:06.291 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-08T08:16:06.293 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-08T08:16:06.296 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-08T08:16:06.297 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-08T08:16:06.538 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-08T08:16:06.551 INFO:teuthology.orchestra.run.smithi116.stdout:==================================================================================================== 2023-11-08T08:16:06.552 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-08T08:16:06.552 INFO:teuthology.orchestra.run.smithi116.stdout:==================================================================================================== 2023-11-08T08:16:06.552 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-08T08:16:06.552 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2023-11-08T08:16:06.553 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:18.0.0-2698.gb820560f.el8 ceph 6.4 k 2023-11-08T08:16:06.553 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:18.0.0-2698.gb820560f.el8 ceph 5.4 M 2023-11-08T08:16:06.553 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:18.0.0-2698.gb820560f.el8 ceph 845 k 2023-11-08T08:16:06.553 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2698.gb820560f.el8 ceph 166 k 2023-11-08T08:16:06.553 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:18.0.0-2698.gb820560f.el8 ceph 1.5 M 2023-11-08T08:16:06.554 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 123 k 2023-11-08T08:16:06.554 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 1.7 M 2023-11-08T08:16:06.554 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 7.4 M 2023-11-08T08:16:06.554 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 47 k 2023-11-08T08:16:06.554 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:18.0.0-2698.gb820560f.el8 ceph 13 M 2023-11-08T08:16:06.555 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:18.0.0-2698.gb820560f.el8 ceph 47 M 2023-11-08T08:16:06.555 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 263 k 2023-11-08T08:16:06.555 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 83 k 2023-11-08T08:16:06.556 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:18.0.0-2698.gb820560f.el8 ceph 258 k 2023-11-08T08:16:06.556 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 18 k 2023-11-08T08:16:06.556 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 baseos 62 k 2023-11-08T08:16:06.558 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2023-11-08T08:16:06.558 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2023-11-08T08:16:06.558 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:18.0.0-2698.gb820560f.el8 ceph 30 k 2023-11-08T08:16:06.559 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 767 k 2023-11-08T08:16:06.559 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:18.0.0-2698.gb820560f.el8 ceph 139 k 2023-11-08T08:16:06.559 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2023-11-08T08:16:06.559 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.32.1-43.el8 baseos 100 k 2023-11-08T08:16:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:1.1.1k-9.el8 baseos 3.2 M 2023-11-08T08:16:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2023-11-08T08:16:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2023-11-08T08:16:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2023-11-08T08:16:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:18.0.0-2698.gb820560f.el8 ceph 190 k 2023-11-08T08:16:06.561 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:18.0.0-2698.gb820560f.el8 ceph 371 k 2023-11-08T08:16:06.561 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:18.0.0-2698.gb820560f.el8 ceph 364 k 2023-11-08T08:16:06.561 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:18.0.0-2698.gb820560f.el8 ceph 107 k 2023-11-08T08:16:06.561 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:18.0.0-2698.gb820560f.el8 ceph 94 k 2023-11-08T08:16:06.561 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:18.0.0-2698.gb820560f.el8 ceph 3.4 M 2023-11-08T08:16:06.562 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:18.0.0-2698.gb820560f.el8 ceph 179 k 2023-11-08T08:16:06.562 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2023-11-08T08:16:06.562 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2023-11-08T08:16:06.562 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-11.el8 baseos 144 k 2023-11-08T08:16:06.562 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2023-11-08T08:16:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.189-3.el8 baseos 76 k 2023-11-08T08:16:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.189-3.el8 baseos 232 k 2023-11-08T08:16:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.189-3.el8 baseos 303 k 2023-11-08T08:16:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2023-11-08T08:16:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2023-11-08T08:16:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2023-11-08T08:16:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.32.1-43.el8 baseos 220 k 2023-11-08T08:16:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.32.1-43.el8 baseos 254 k 2023-11-08T08:16:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.32.1-43.el8 baseos 236 k 2023-11-08T08:16:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 3.7 M 2023-11-08T08:16:06.565 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 3.6 M 2023-11-08T08:16:06.565 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.32.1-43.el8 baseos 179 k 2023-11-08T08:16:06.565 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.32.1-43.el8 baseos 99 k 2023-11-08T08:16:06.565 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.32.1-43.el8 baseos 2.5 M 2023-11-08T08:16:06.565 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.32.1-43.el8 baseos 103 k 2023-11-08T08:16:06.566 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2023-11-08T08:16:06.566 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2023-11-08T08:16:06.566 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-08T08:16:06.566 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2023-11-08T08:16:06.566 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:18.0.0-2698.gb820560f.el8 ceph 22 M 2023-11-08T08:16:06.567 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 23 k 2023-11-08T08:16:06.567 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:18.0.0-2698.gb820560f.el8 ceph 2.3 M 2023-11-08T08:16:06.567 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 236 k 2023-11-08T08:16:06.567 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:18.0.0-2698.gb820560f.el8 ceph 4.6 M 2023-11-08T08:16:06.567 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:18.0.0-2698.gb820560f.el8 ceph 18 M 2023-11-08T08:16:06.568 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2698.gb820560f.el8 ceph-noarch 15 k 2023-11-08T08:16:06.568 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:18.0.0-2698.gb820560f.el8 ceph 24 k 2023-11-08T08:16:06.568 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-08T08:16:06.568 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-08T08:16:06.568 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-08T08:16:06.569 INFO:teuthology.orchestra.run.smithi116.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2023-11-08T08:16:06.569 INFO:teuthology.orchestra.run.smithi116.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2023-11-08T08:16:06.569 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:18.0.0-2698.gb820560f.el8 ceph 137 k 2023-11-08T08:16:06.569 INFO:teuthology.orchestra.run.smithi116.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2023-11-08T08:16:06.569 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-08T08:16:06.570 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-08T08:16:06.570 INFO:teuthology.orchestra.run.smithi116.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2023-11-08T08:16:06.570 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-08T08:16:06.570 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-08T08:16:06.570 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 474 k 2023-11-08T08:16:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-08T08:16:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:18.0.0-2698.gb820560f.el8 ceph 6.5 M 2023-11-08T08:16:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2023-11-08T08:16:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2023-11-08T08:16:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-08T08:16:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2023-11-08T08:16:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2023-11-08T08:16:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-08T08:16:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-08T08:16:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-08T08:16:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-08T08:16:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2023-11-08T08:16:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2023-11-08T08:16:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2023-11-08T08:16:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-08T08:16:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-08T08:16:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-08T08:16:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-08T08:16:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-08T08:16:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:18.0.0-2698.gb820560f.el8 ceph 45 k 2023-11-08T08:16:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:18.0.0-2698.gb820560f.el8 ceph 117 k 2023-11-08T08:16:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-08T08:16:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-08T08:16:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-08T08:16:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-08T08:16:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-08T08:16:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-08T08:16:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-08T08:16:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-08T08:16:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-08T08:16:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-08T08:16:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-08T08:16:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-08T08:16:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-08T08:16:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-08T08:16:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-08T08:16:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-08T08:16:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-08T08:16:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-08T08:16:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-08T08:16:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-08T08:16:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-08T08:16:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-08T08:16:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-08T08:16:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-08T08:16:06.580 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-08T08:16:06.580 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-08T08:16:06.580 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-08T08:16:06.580 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-08T08:16:06.580 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-08T08:16:06.581 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-08T08:16:06.581 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-08T08:16:06.581 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-08T08:16:06.581 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-08T08:16:06.581 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-08T08:16:06.582 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-08T08:16:06.582 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-08T08:16:06.582 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-08T08:16:06.582 INFO:teuthology.orchestra.run.smithi116.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-08T08:16:06.582 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-08T08:16:06.583 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-08T08:16:06.583 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-08T08:16:06.583 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-08T08:16:06.583 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-08T08:16:06.583 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-08T08:16:06.584 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-08T08:16:06.584 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-08T08:16:06.584 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2023-11-08T08:16:06.584 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-08T08:16:06.584 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2023-11-08T08:16:06.585 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling module streams: 2023-11-08T08:16:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql 10 2023-11-08T08:16:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:16:06.585 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-08T08:16:06.585 INFO:teuthology.orchestra.run.smithi116.stdout:==================================================================================================== 2023-11-08T08:16:06.586 INFO:teuthology.orchestra.run.smithi116.stdout:Install 128 Packages 2023-11-08T08:16:06.586 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 19 Packages 2023-11-08T08:16:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:16:06.587 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 210 M 2023-11-08T08:16:06.587 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-08T08:16:07.209 INFO:teuthology.orchestra.run.smithi116.stdout:(1/147): ceph-18.0.0-2698.gb820560f.el8.x86_64. 50 kB/s | 6.4 kB 00:00 2023-11-08T08:16:07.435 INFO:teuthology.orchestra.run.smithi116.stdout:(2/147): ceph-fuse-18.0.0-2698.gb820560f.el8.x8 3.7 MB/s | 845 kB 00:00 2023-11-08T08:16:07.501 INFO:teuthology.orchestra.run.smithi116.stdout:(3/147): ceph-immutable-object-cache-18.0.0-269 2.4 MB/s | 166 kB 00:00 2023-11-08T08:16:07.968 INFO:teuthology.orchestra.run.smithi116.stdout:(4/147): ceph-mds-18.0.0-2698.gb820560f.el8.x86 4.9 MB/s | 2.3 MB 00:00 2023-11-08T08:16:08.419 INFO:teuthology.orchestra.run.smithi116.stdout:(5/147): ceph-common-18.0.0-2698.gb820560f.el8. 16 MB/s | 22 MB 00:01 2023-11-08T08:16:08.461 INFO:teuthology.orchestra.run.smithi116.stdout:(6/147): ceph-mgr-18.0.0-2698.gb820560f.el8.x86 3.1 MB/s | 1.5 MB 00:00 2023-11-08T08:16:08.927 INFO:teuthology.orchestra.run.smithi116.stdout:(7/147): ceph-base-18.0.0-2698.gb820560f.el8.x8 2.9 MB/s | 5.4 MB 00:01 2023-11-08T08:16:08.994 INFO:teuthology.orchestra.run.smithi116.stdout:(8/147): ceph-mon-18.0.0-2698.gb820560f.el8.x86 8.0 MB/s | 4.6 MB 00:00 2023-11-08T08:16:09.044 INFO:teuthology.orchestra.run.smithi116.stdout:(9/147): ceph-selinux-18.0.0-2698.gb820560f.el8 489 kB/s | 24 kB 00:00 2023-11-08T08:16:10.370 INFO:teuthology.orchestra.run.smithi116.stdout:(10/147): ceph-radosgw-18.0.0-2698.gb820560f.el 9.0 MB/s | 13 MB 00:01 2023-11-08T08:16:10.620 INFO:teuthology.orchestra.run.smithi116.stdout:(11/147): ceph-osd-18.0.0-2698.gb820560f.el8.x8 8.2 MB/s | 18 MB 00:02 2023-11-08T08:16:10.645 INFO:teuthology.orchestra.run.smithi116.stdout:(12/147): cephfs-mirror-18.0.0-2698.gb820560f.e 939 kB/s | 258 kB 00:00 2023-11-08T08:16:10.679 INFO:teuthology.orchestra.run.smithi116.stdout:(13/147): libcephfs-devel-18.0.0-2698.gb820560f 523 kB/s | 30 kB 00:00 2023-11-08T08:16:10.746 INFO:teuthology.orchestra.run.smithi116.stdout:(14/147): libcephsqlite-18.0.0-2698.gb820560f.e 2.0 MB/s | 137 kB 00:00 2023-11-08T08:16:10.780 INFO:teuthology.orchestra.run.smithi116.stdout:(15/147): libcephfs2-18.0.0-2698.gb820560f.el8. 5.6 MB/s | 767 kB 00:00 2023-11-08T08:16:10.805 INFO:teuthology.orchestra.run.smithi116.stdout:(16/147): librados-devel-18.0.0-2698.gb820560f. 2.3 MB/s | 139 kB 00:00 2023-11-08T08:16:10.872 INFO:teuthology.orchestra.run.smithi116.stdout:(17/147): libradosstriper1-18.0.0-2698.gb820560 5.0 MB/s | 474 kB 00:00 2023-11-08T08:16:10.955 INFO:teuthology.orchestra.run.smithi116.stdout:(18/147): python3-ceph-argparse-18.0.0-2698.gb8 535 kB/s | 45 kB 00:00 2023-11-08T08:16:11.030 INFO:teuthology.orchestra.run.smithi116.stdout:(19/147): python3-ceph-common-18.0.0-2698.gb820 1.5 MB/s | 117 kB 00:00 2023-11-08T08:16:11.123 INFO:teuthology.orchestra.run.smithi116.stdout:(20/147): python3-cephfs-18.0.0-2698.gb820560f. 2.0 MB/s | 190 kB 00:00 2023-11-08T08:16:11.231 INFO:teuthology.orchestra.run.smithi116.stdout:(21/147): python3-rados-18.0.0-2698.gb820560f.e 3.3 MB/s | 371 kB 00:00 2023-11-08T08:16:11.340 INFO:teuthology.orchestra.run.smithi116.stdout:(22/147): python3-rbd-18.0.0-2698.gb820560f.el8 3.3 MB/s | 364 kB 00:00 2023-11-08T08:16:11.432 INFO:teuthology.orchestra.run.smithi116.stdout:(23/147): python3-rgw-18.0.0-2698.gb820560f.el8 1.1 MB/s | 107 kB 00:00 2023-11-08T08:16:11.524 INFO:teuthology.orchestra.run.smithi116.stdout:(24/147): rbd-fuse-18.0.0-2698.gb820560f.el8.x8 1.0 MB/s | 94 kB 00:00 2023-11-08T08:16:11.708 INFO:teuthology.orchestra.run.smithi116.stdout:(25/147): librgw2-18.0.0-2698.gb820560f.el8.x86 7.2 MB/s | 6.5 MB 00:00 2023-11-08T08:16:11.791 INFO:teuthology.orchestra.run.smithi116.stdout:(26/147): rbd-nbd-18.0.0-2698.gb820560f.el8.x86 2.1 MB/s | 179 kB 00:00 2023-11-08T08:16:11.866 INFO:teuthology.orchestra.run.smithi116.stdout:(27/147): ceph-grafana-dashboards-18.0.0-2698.g 309 kB/s | 23 kB 00:00 2023-11-08T08:16:11.925 INFO:teuthology.orchestra.run.smithi116.stdout:(28/147): rbd-mirror-18.0.0-2698.gb820560f.el8. 8.6 MB/s | 3.4 MB 00:00 2023-11-08T08:16:11.950 INFO:teuthology.orchestra.run.smithi116.stdout:(29/147): ceph-mgr-cephadm-18.0.0-2698.gb820560 1.4 MB/s | 123 kB 00:00 2023-11-08T08:16:12.134 INFO:teuthology.orchestra.run.smithi116.stdout:(30/147): ceph-mgr-dashboard-18.0.0-2698.gb8205 8.3 MB/s | 1.7 MB 00:00 2023-11-08T08:16:12.226 INFO:teuthology.orchestra.run.smithi116.stdout:(31/147): ceph-mgr-modules-core-18.0.0-2698.gb8 2.5 MB/s | 236 kB 00:00 2023-11-08T08:16:12.618 INFO:teuthology.orchestra.run.smithi116.stdout:(32/147): ceph-test-18.0.0-2698.gb820560f.el8.x 13 MB/s | 47 MB 00:03 2023-11-08T08:16:12.635 INFO:teuthology.orchestra.run.smithi116.stdout:(33/147): ceph-mgr-rook-18.0.0-2698.gb820560f.e 116 kB/s | 47 kB 00:00 2023-11-08T08:16:12.668 INFO:teuthology.orchestra.run.smithi116.stdout:(34/147): ceph-prometheus-alerts-18.0.0-2698.gb 292 kB/s | 15 kB 00:00 2023-11-08T08:16:12.702 INFO:teuthology.orchestra.run.smithi116.stdout:(35/147): ceph-volume-18.0.0-2698.gb820560f.el8 3.8 MB/s | 263 kB 00:00 2023-11-08T08:16:12.727 INFO:teuthology.orchestra.run.smithi116.stdout:(36/147): cephadm-18.0.0-2698.gb820560f.el8.noa 1.4 MB/s | 83 kB 00:00 2023-11-08T08:16:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:(37/147): cephfs-top-18.0.0-2698.gb820560f.el8. 270 kB/s | 18 kB 00:00 2023-11-08T08:16:12.878 INFO:teuthology.orchestra.run.smithi116.stdout:(38/147): ceph-mgr-diskprediction-local-18.0.0- 8.0 MB/s | 7.4 MB 00:00 2023-11-08T08:16:12.937 INFO:teuthology.orchestra.run.smithi116.stdout:(39/147): automake-1.16.1-8.el8.noarch.rpm 3.5 MB/s | 740 kB 00:00 2023-11-08T08:16:12.979 INFO:teuthology.orchestra.run.smithi116.stdout:(40/147): jq-1.6-8.el8.x86_64.rpm 4.8 MB/s | 203 kB 00:00 2023-11-08T08:16:13.012 INFO:teuthology.orchestra.run.smithi116.stdout:(41/147): bison-3.0.4-10.el8.x86_64.rpm 2.8 MB/s | 688 kB 00:00 2023-11-08T08:16:13.037 INFO:teuthology.orchestra.run.smithi116.stdout:(42/147): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-11-08T08:16:13.063 INFO:teuthology.orchestra.run.smithi116.stdout:(43/147): flex-2.6.1-9.el8.x86_64.rpm 1.7 MB/s | 320 kB 00:00 2023-11-08T08:16:13.088 INFO:teuthology.orchestra.run.smithi116.stdout:(44/147): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2023-11-08T08:16:13.114 INFO:teuthology.orchestra.run.smithi116.stdout:(45/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-11-08T08:16:13.147 INFO:teuthology.orchestra.run.smithi116.stdout:(46/147): libtool-2.4.6-25.el8.x86_64.rpm 6.3 MB/s | 709 kB 00:00 2023-11-08T08:16:13.172 INFO:teuthology.orchestra.run.smithi116.stdout:(47/147): oniguruma-6.8.2-2.el8.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2023-11-08T08:16:13.281 INFO:teuthology.orchestra.run.smithi116.stdout:(48/147): postgresql-10.23-2.module_el8+572+929 14 MB/s | 1.6 MB 00:00 2023-11-08T08:16:13.373 INFO:teuthology.orchestra.run.smithi116.stdout:(49/147): postgresql-contrib-10.23-2.module_el8 9.6 MB/s | 901 kB 00:00 2023-11-08T08:16:13.457 INFO:teuthology.orchestra.run.smithi116.stdout:(50/147): openblas-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-11-08T08:16:13.557 INFO:teuthology.orchestra.run.smithi116.stdout:(51/147): openblas-threads-0.3.15-4.el8.x86_64. 12 MB/s | 4.9 MB 00:00 2023-11-08T08:16:13.607 INFO:teuthology.orchestra.run.smithi116.stdout:(52/147): python3-jinja2-2.10.1-3.el8.noarch.rp 11 MB/s | 538 kB 00:00 2023-11-08T08:16:13.641 INFO:teuthology.orchestra.run.smithi116.stdout:(53/147): python3-jsonpatch-1.21-2.el8.noarch.r 798 kB/s | 27 kB 00:00 2023-11-08T08:16:13.674 INFO:teuthology.orchestra.run.smithi116.stdout:(54/147): python3-jsonpointer-1.10-11.el8.noarc 598 kB/s | 20 kB 00:00 2023-11-08T08:16:13.716 INFO:teuthology.orchestra.run.smithi116.stdout:(55/147): python3-mako-1.0.6-14.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-11-08T08:16:13.751 INFO:teuthology.orchestra.run.smithi116.stdout:(56/147): python3-markupsafe-0.23-19.el8.x86_64 1.1 MB/s | 39 kB 00:00 2023-11-08T08:16:13.826 INFO:teuthology.orchestra.run.smithi116.stdout:(57/147): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2023-11-08T08:16:13.892 INFO:teuthology.orchestra.run.smithi116.stdout:(58/147): postgresql-server-10.23-2.module_el8+ 9.9 MB/s | 5.1 MB 00:00 2023-11-08T08:16:13.917 INFO:teuthology.orchestra.run.smithi116.stdout:(59/147): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.4 MB/s | 225 kB 00:00 2023-11-08T08:16:13.934 INFO:teuthology.orchestra.run.smithi116.stdout:(60/147): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-11-08T08:16:13.960 INFO:teuthology.orchestra.run.smithi116.stdout:(61/147): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2023-11-08T08:16:13.985 INFO:teuthology.orchestra.run.smithi116.stdout:(62/147): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-11-08T08:16:14.010 INFO:teuthology.orchestra.run.smithi116.stdout:(63/147): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-11-08T08:16:14.027 INFO:teuthology.orchestra.run.smithi116.stdout:(64/147): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2023-11-08T08:16:14.119 INFO:teuthology.orchestra.run.smithi116.stdout:(65/147): python3-numpy-1.14.3-10.el8.x86_64.rp 10 MB/s | 3.7 MB 00:00 2023-11-08T08:16:14.153 INFO:teuthology.orchestra.run.smithi116.stdout:(66/147): python3-werkzeug-0.12.2-4.el8.noarch. 3.6 MB/s | 457 kB 00:00 2023-11-08T08:16:14.178 INFO:teuthology.orchestra.run.smithi116.stdout:(67/147): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-11-08T08:16:14.203 INFO:teuthology.orchestra.run.smithi116.stdout:(68/147): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-11-08T08:16:14.228 INFO:teuthology.orchestra.run.smithi116.stdout:(69/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2023-11-08T08:16:14.480 INFO:teuthology.orchestra.run.smithi116.stdout:(70/147): python3-scipy-1.0.0-21.module_el8.5.0 29 MB/s | 14 MB 00:00 2023-11-08T08:16:14.504 INFO:teuthology.orchestra.run.smithi116.stdout:(71/147): elfutils-libelf-devel-0.189-3.el8.x86 205 kB/s | 62 kB 00:00 2023-11-08T08:16:14.529 INFO:teuthology.orchestra.run.smithi116.stdout:(72/147): fuse-2.9.7-17.el8.x86_64.rpm 276 kB/s | 83 kB 00:00 2023-11-08T08:16:14.588 INFO:teuthology.orchestra.run.smithi116.stdout:(73/147): libaio-devel-0.3.112-1.el8.x86_64.rpm 320 kB/s | 19 kB 00:00 2023-11-08T08:16:14.663 INFO:teuthology.orchestra.run.smithi116.stdout:(74/147): libcom_err-devel-1.45.6-5.el8.x86_64. 517 kB/s | 39 kB 00:00 2023-11-08T08:16:14.689 INFO:teuthology.orchestra.run.smithi116.stdout:(75/147): keyutils-libs-devel-1.5.10-9.el8.x86_ 230 kB/s | 48 kB 00:00 2023-11-08T08:16:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:(76/147): krb5-devel-1.18.2-22.el8.x86_64.rpm 2.6 MB/s | 561 kB 00:00 2023-11-08T08:16:14.772 INFO:teuthology.orchestra.run.smithi116.stdout:(77/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 805 kB/s | 47 kB 00:00 2023-11-08T08:16:14.814 INFO:teuthology.orchestra.run.smithi116.stdout:(78/147): libgfortran-8.5.0-21.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2023-11-08T08:16:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:(79/147): libquadmath-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2023-11-08T08:16:14.873 INFO:teuthology.orchestra.run.smithi116.stdout:(80/147): libselinux-devel-2.9-8.el8.x86_64.rpm 1.9 MB/s | 200 kB 00:00 2023-11-08T08:16:14.899 INFO:teuthology.orchestra.run.smithi116.stdout:(81/147): libsepol-devel-2.9-3.el8.x86_64.rpm 1.0 MB/s | 87 kB 00:00 2023-11-08T08:16:14.924 INFO:teuthology.orchestra.run.smithi116.stdout:(82/147): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 100 kB 00:00 2023-11-08T08:16:14.949 INFO:teuthology.orchestra.run.smithi116.stdout:(83/147): libverto-devel-0.3.2-2.el8.x86_64.rpm 238 kB/s | 18 kB 00:00 2023-11-08T08:16:14.974 INFO:teuthology.orchestra.run.smithi116.stdout:(84/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 581 kB/s | 44 kB 00:00 2023-11-08T08:16:15.024 INFO:teuthology.orchestra.run.smithi116.stdout:(85/147): pcre2-devel-10.32-3.el8.x86_64.rpm 7.9 MB/s | 605 kB 00:00 2023-11-08T08:16:15.050 INFO:teuthology.orchestra.run.smithi116.stdout:(86/147): pcre2-utf16-10.32-3.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2023-11-08T08:16:15.093 INFO:teuthology.orchestra.run.smithi116.stdout:(87/147): pcre2-utf32-10.32-3.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2023-11-08T08:16:15.117 INFO:teuthology.orchestra.run.smithi116.stdout:(88/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 3.6 MB/s | 249 kB 00:00 2023-11-08T08:16:15.159 INFO:teuthology.orchestra.run.smithi116.stdout:(89/147): python3-cryptography-3.2.1-6.el8.x86_ 9.4 MB/s | 641 kB 00:00 2023-11-08T08:16:15.184 INFO:teuthology.orchestra.run.smithi116.stdout:(90/147): python3-jwt-1.6.1-2.el8.noarch.rpm 636 kB/s | 43 kB 00:00 2023-11-08T08:16:15.260 INFO:teuthology.orchestra.run.smithi116.stdout:(91/147): openssl-devel-1.1.1k-9.el8.x86_64.rpm 9.5 MB/s | 3.2 MB 00:00 2023-11-08T08:16:15.285 INFO:teuthology.orchestra.run.smithi116.stdout:(92/147): python3-pycparser-2.14-14.el8.noarch. 1.1 MB/s | 109 kB 00:00 2023-11-08T08:16:15.310 INFO:teuthology.orchestra.run.smithi116.stdout:(93/147): python3-oauthlib-2.1.0-1.el8.noarch.r 1.0 MB/s | 155 kB 00:00 2023-11-08T08:16:15.335 INFO:teuthology.orchestra.run.smithi116.stdout:(94/147): python3-requests-oauthlib-1.0.0-1.el8 566 kB/s | 43 kB 00:00 2023-11-08T08:16:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:(95/147): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 986 kB/s | 74 kB 00:00 2023-11-08T08:16:15.386 INFO:teuthology.orchestra.run.smithi116.stdout:(96/147): zlib-devel-1.2.11-21.el8.x86_64.rpm 768 kB/s | 58 kB 00:00 2023-11-08T08:16:15.469 INFO:teuthology.orchestra.run.smithi116.stdout:(97/147): liboath-2.6.2-3.el8.x86_64.rpm 542 kB/s | 59 kB 00:00 2023-11-08T08:16:15.503 INFO:teuthology.orchestra.run.smithi116.stdout:(98/147): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-11-08T08:16:15.528 INFO:teuthology.orchestra.run.smithi116.stdout:(99/147): libunwind-1.3.1-3.el8.x86_64.rpm 564 kB/s | 75 kB 00:00 2023-11-08T08:16:15.553 INFO:teuthology.orchestra.run.smithi116.stdout:(100/147): python3-bcrypt-3.1.6-2.el8.1.x86_64. 871 kB/s | 44 kB 00:00 2023-11-08T08:16:15.579 INFO:teuthology.orchestra.run.smithi116.stdout:(101/147): python3-asyncssh-2.7.0-2.el8.noarch. 4.3 MB/s | 476 kB 00:00 2023-11-08T08:16:15.604 INFO:teuthology.orchestra.run.smithi116.stdout:(102/147): python3-cachetools-3.1.1-4.el8.noarc 657 kB/s | 33 kB 00:00 2023-11-08T08:16:15.629 INFO:teuthology.orchestra.run.smithi116.stdout:(103/147): python3-certifi-2018.10.15-7.el8.noa 325 kB/s | 16 kB 00:00 2023-11-08T08:16:15.655 INFO:teuthology.orchestra.run.smithi116.stdout:(104/147): python3-beautifulsoup4-4.6.3-2.el8.1 1.4 MB/s | 185 kB 00:00 2023-11-08T08:16:15.680 INFO:teuthology.orchestra.run.smithi116.stdout:(105/147): python3-cheroot-8.5.2-1.el8.noarch.r 2.2 MB/s | 173 kB 00:00 2023-11-08T08:16:15.706 INFO:teuthology.orchestra.run.smithi116.stdout:(106/147): python3-google-auth-1.1.1-10.el8.noa 1.6 MB/s | 81 kB 00:00 2023-11-08T08:16:15.730 INFO:teuthology.orchestra.run.smithi116.stdout:(107/147): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-11-08T08:16:15.756 INFO:teuthology.orchestra.run.smithi116.stdout:(108/147): python3-jaraco-functools-2.0-4.el8.n 351 kB/s | 18 kB 00:00 2023-11-08T08:16:15.814 INFO:teuthology.orchestra.run.smithi116.stdout:(109/147): python3-cherrypy-18.4.0-1.el8.noarch 2.0 MB/s | 384 kB 00:00 2023-11-08T08:16:15.848 INFO:teuthology.orchestra.run.smithi116.stdout:(110/147): python3-logutils-0.3.5-11.el8.noarch 530 kB/s | 49 kB 00:00 2023-11-08T08:16:15.874 INFO:teuthology.orchestra.run.smithi116.stdout:(111/147): python3-more-itertools-7.2.0-3.el8.n 986 kB/s | 59 kB 00:00 2023-11-08T08:16:15.899 INFO:teuthology.orchestra.run.smithi116.stdout:(112/147): python3-natsort-7.1.1-2.el8.noarch.r 1.2 MB/s | 60 kB 00:00 2023-11-08T08:16:15.925 INFO:teuthology.orchestra.run.smithi116.stdout:(113/147): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-11-08T08:16:15.949 INFO:teuthology.orchestra.run.smithi116.stdout:(114/147): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-11-08T08:16:15.975 INFO:teuthology.orchestra.run.smithi116.stdout:(115/147): python3-repoze-lru-0.7-6.el8.noarch. 672 kB/s | 34 kB 00:00 2023-11-08T08:16:16.000 INFO:teuthology.orchestra.run.smithi116.stdout:(116/147): python3-routes-2.4.1-12.el8.noarch.r 3.8 MB/s | 196 kB 00:00 2023-11-08T08:16:16.025 INFO:teuthology.orchestra.run.smithi116.stdout:(117/147): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-11-08T08:16:16.050 INFO:teuthology.orchestra.run.smithi116.stdout:(118/147): python3-simplegeneric-0.8.1-17.el8.n 375 kB/s | 19 kB 00:00 2023-11-08T08:16:16.075 INFO:teuthology.orchestra.run.smithi116.stdout:(119/147): python3-singledispatch-3.4.0.3-18.el 477 kB/s | 24 kB 00:00 2023-11-08T08:16:16.101 INFO:teuthology.orchestra.run.smithi116.stdout:(120/147): python3-tempora-1.14.1-5.el8.noarch. 584 kB/s | 29 kB 00:00 2023-11-08T08:16:16.126 INFO:teuthology.orchestra.run.smithi116.stdout:(121/147): python3-trustme-0.6.0-4.el8.noarch.r 549 kB/s | 27 kB 00:00 2023-11-08T08:16:16.160 INFO:teuthology.orchestra.run.smithi116.stdout:(122/147): python3-kubernetes-11.0.0-6.el8.noar 2.7 MB/s | 1.2 MB 00:00 2023-11-08T08:16:16.185 INFO:teuthology.orchestra.run.smithi116.stdout:(123/147): python3-waitress-1.4.3-1.el8.noarch. 3.0 MB/s | 255 kB 00:00 2023-11-08T08:16:16.202 INFO:teuthology.orchestra.run.smithi116.stdout:(124/147): python3-websocket-client-0.56.0-5.el 1.4 MB/s | 61 kB 00:00 2023-11-08T08:16:16.227 INFO:teuthology.orchestra.run.smithi116.stdout:(125/147): python3-webob-1.8.5-1.el8.1.noarch.r 2.4 MB/s | 251 kB 00:00 2023-11-08T08:16:16.244 INFO:teuthology.orchestra.run.smithi116.stdout:(126/147): python3-zc-lockfile-2.0-2.el8.noarch 550 kB/s | 23 kB 00:00 2023-11-08T08:16:16.269 INFO:teuthology.orchestra.run.smithi116.stdout:(127/147): python3-webtest-2.0.33-1.el8.noarch. 1.0 MB/s | 85 kB 00:00 2023-11-08T08:16:16.374 INFO:teuthology.orchestra.run.smithi116.stdout:(128/147): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2023-11-08T08:16:16.520 INFO:teuthology.orchestra.run.smithi116.stdout:(129/147): NetworkManager-initscripts-updown-1. 991 kB/s | 144 kB 00:00 2023-11-08T08:16:16.578 INFO:teuthology.orchestra.run.smithi116.stdout:(130/147): acl-2.2.53-3.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-11-08T08:16:16.629 INFO:teuthology.orchestra.run.smithi116.stdout:(131/147): librados2-18.0.0-2698.gb820560f.el8. 9.5 MB/s | 3.7 MB 00:00 2023-11-08T08:16:16.654 INFO:teuthology.orchestra.run.smithi116.stdout:(132/147): elfutils-debuginfod-client-0.189-3.e 1.0 MB/s | 76 kB 00:00 2023-11-08T08:16:16.779 INFO:teuthology.orchestra.run.smithi116.stdout:(133/147): librbd1-18.0.0-2698.gb820560f.el8.x8 7.0 MB/s | 3.6 MB 00:00 2023-11-08T08:16:16.805 INFO:teuthology.orchestra.run.smithi116.stdout:(134/147): elfutils-libs-0.189-3.el8.x86_64.rpm 2.0 MB/s | 303 kB 00:00 2023-11-08T08:16:16.872 INFO:teuthology.orchestra.run.smithi116.stdout:(135/147): elfutils-libelf-0.189-3.el8.x86_64.r 957 kB/s | 232 kB 00:00 2023-11-08T08:16:16.897 INFO:teuthology.orchestra.run.smithi116.stdout:(136/147): libacl-2.2.53-3.el8.x86_64.rpm 379 kB/s | 35 kB 00:00 2023-11-08T08:16:16.980 INFO:teuthology.orchestra.run.smithi116.stdout:(137/147): libblkid-2.32.1-43.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2023-11-08T08:16:17.006 INFO:teuthology.orchestra.run.smithi116.stdout:(138/147): libacl-devel-2.2.53-3.el8.x86_64.rpm 634 kB/s | 85 kB 00:00 2023-11-08T08:16:17.048 INFO:teuthology.orchestra.run.smithi116.stdout:(139/147): iproute-6.2.0-5.el8.x86_64.rpm 3.1 MB/s | 854 kB 00:00 2023-11-08T08:16:17.073 INFO:teuthology.orchestra.run.smithi116.stdout:(140/147): libfdisk-2.32.1-43.el8.x86_64.rpm 2.7 MB/s | 254 kB 00:00 2023-11-08T08:16:17.115 INFO:teuthology.orchestra.run.smithi116.stdout:(141/147): libsmartcols-2.32.1-43.el8.x86_64.rp 2.6 MB/s | 179 kB 00:00 2023-11-08T08:16:17.140 INFO:teuthology.orchestra.run.smithi116.stdout:(142/147): libuuid-2.32.1-43.el8.x86_64.rpm 1.4 MB/s | 99 kB 00:00 2023-11-08T08:16:17.166 INFO:teuthology.orchestra.run.smithi116.stdout:(143/147): libmount-2.32.1-43.el8.x86_64.rpm 1.4 MB/s | 236 kB 00:00 2023-11-08T08:16:17.199 INFO:teuthology.orchestra.run.smithi116.stdout:(144/147): util-linux-user-2.32.1-43.el8.x86_64 1.7 MB/s | 103 kB 00:00 2023-11-08T08:16:17.299 INFO:teuthology.orchestra.run.smithi116.stdout:(145/147): util-linux-2.32.1-43.el8.x86_64.rpm 13 MB/s | 2.5 MB 00:00 2023-11-08T08:16:17.333 INFO:teuthology.orchestra.run.smithi116.stdout:(146/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.1 MB/s | 1.1 MB 00:00 2023-11-08T08:16:17.391 INFO:teuthology.orchestra.run.smithi116.stdout:(147/147): xfsdump-3.1.8-7.el8.x86_64.rpm 1.4 MB/s | 334 kB 00:00 2023-11-08T08:16:17.397 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-08T08:16:17.398 INFO:teuthology.orchestra.run.smithi116.stdout:Total 19 MB/s | 210 MB 00:10 2023-11-08T08:16:18.319 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-08T08:16:18.754 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-08T08:16:18.754 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-08T08:16:20.194 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-08T08:16:20.197 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-08T08:16:25.792 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-08T08:16:25.996 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: thrift-0.13.0-2.el8.x86_64 1/1 2023-11-08T08:16:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/166 2023-11-08T08:16:26.351 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 2/166 2023-11-08T08:16:26.435 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 2/166 2023-11-08T08:16:26.676 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 3/166 2023-11-08T08:16:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 3/166 2023-11-08T08:16:26.944 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/166 2023-11-08T08:16:27.114 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 5/166 2023-11-08T08:16:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/166 2023-11-08T08:16:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/166 2023-11-08T08:16:27.886 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/166 2023-11-08T08:16:28.059 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:18.0.0-2698.gb820560f.el8.x86_64 8/166 2023-11-08T08:16:28.448 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:18.0.0-2698.gb820560f.el8.x86_64 8/166 2023-11-08T08:16:28.634 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 9/166 2023-11-08T08:16:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 9/166 2023-11-08T08:16:28.925 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 10/166 2023-11-08T08:16:29.022 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 10/166 2023-11-08T08:16:29.232 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:18.0.0-2698.gb820560f.el8.x86_64 11/166 2023-11-08T08:16:29.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 12/166 2023-11-08T08:16:29.585 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 12/166 2023-11-08T08:16:29.817 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/166 2023-11-08T08:16:29.887 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/166 2023-11-08T08:16:30.114 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 14/166 2023-11-08T08:16:30.340 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 15/166 2023-11-08T08:16:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpq-13.5-1.el8.x86_64 16/166 2023-11-08T08:16:30.937 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 17/166 2023-11-08T08:16:31.075 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 18/166 2023-11-08T08:16:31.182 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 18/166 2023-11-08T08:16:31.342 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86 19/166 2023-11-08T08:16:31.433 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86 19/166 2023-11-08T08:16:31.648 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 20/166 2023-11-08T08:16:31.906 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 21/166 2023-11-08T08:16:32.122 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 22/166 2023-11-08T08:16:32.259 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 23/166 2023-11-08T08:16:32.394 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 23/166 2023-11-08T08:16:32.525 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 24/166 2023-11-08T08:16:33.128 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 24/166 2023-11-08T08:16:33.350 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.15-4.el8.x86_64 25/166 2023-11-08T08:16:33.956 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 25/166 2023-11-08T08:16:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 26/166 2023-11-08T08:16:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 26/166 2023-11-08T08:16:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 27/166 2023-11-08T08:16:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 28/166 2023-11-08T08:16:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 29/166 2023-11-08T08:16:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 30/166 2023-11-08T08:16:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:18.0.0-2698.gb820560f.el8. 31/166 2023-11-08T08:16:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:18.0.0-2698.gb820560f.el 32/166 2023-11-08T08:16:36.243 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:18.0.0-2698.gb820560f.el8.x86_6 33/166 2023-11-08T08:16:37.190 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/166 2023-11-08T08:16:37.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/166 2023-11-08T08:16:37.755 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/166 2023-11-08T08:16:37.908 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/166 2023-11-08T08:16:39.691 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/166 2023-11-08T08:16:39.961 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 38/166 2023-11-08T08:16:40.183 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 39/166 2023-11-08T08:16:40.384 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/166 2023-11-08T08:16:40.547 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:18.0.0-2698.gb820560f.el8.x86_64 41/166 2023-11-08T08:16:40.748 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:18.0.0-2698.gb820560f.el8.x86_6 42/166 2023-11-08T08:16:40.918 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 43/166 2023-11-08T08:16:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.189-3.el8.x86_64 44/166 2023-11-08T08:16:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 45/166 2023-11-08T08:16:41.407 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 45/166 2023-11-08T08:16:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 46/166 2023-11-08T08:16:41.633 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 46/166 2023-11-08T08:16:41.819 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/166 2023-11-08T08:16:41.982 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 48/166 2023-11-08T08:16:42.471 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 48/166 2023-11-08T08:16:42.716 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 49/166 2023-11-08T08:16:42.730 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 49/166 2023-11-08T08:16:42.901 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 50/166 2023-11-08T08:16:43.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 51/166 2023-11-08T08:16:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 52/166 2023-11-08T08:16:43.521 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 53/166 2023-11-08T08:16:43.684 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 54/166 2023-11-08T08:16:43.864 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 55/166 2023-11-08T08:16:44.059 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 56/166 2023-11-08T08:16:44.275 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 57/166 2023-11-08T08:16:44.418 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 58/166 2023-11-08T08:16:44.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 59/166 2023-11-08T08:16:44.712 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 60/166 2023-11-08T08:16:44.863 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 61/166 2023-11-08T08:16:45.026 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-2.6-1.el8.noarch 62/166 2023-11-08T08:16:45.182 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 63/166 2023-11-08T08:16:45.418 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 64/166 2023-11-08T08:16:45.649 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 65/166 2023-11-08T08:16:45.885 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 66/166 2023-11-08T08:16:46.218 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 67/166 2023-11-08T08:16:46.457 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 68/166 2023-11-08T08:16:46.648 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 69/166 2023-11-08T08:16:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 70/166 2023-11-08T08:16:47.648 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 71/166 2023-11-08T08:16:47.722 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 71/166 2023-11-08T08:16:47.895 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:18.0.0-2698.gb820560f.el8.x86_64 72/166 2023-11-08T08:16:48.169 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 73/166 2023-11-08T08:16:48.522 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 74/166 2023-11-08T08:16:48.842 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 75/166 2023-11-08T08:16:49.033 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 76/166 2023-11-08T08:16:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 77/166 2023-11-08T08:16:49.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 78/166 2023-11-08T08:16:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 79/166 2023-11-08T08:16:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 80/166 2023-11-08T08:16:51.064 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 81/166 2023-11-08T08:16:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 82/166 2023-11-08T08:16:51.529 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 83/166 2023-11-08T08:16:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 84/166 2023-11-08T08:16:52.052 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 85/166 2023-11-08T08:16:52.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 86/166 2023-11-08T08:16:52.547 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 87/166 2023-11-08T08:16:52.786 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 88/166 2023-11-08T08:16:52.933 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 89/166 2023-11-08T08:16:53.103 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 90/166 2023-11-08T08:16:53.418 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 91/166 2023-11-08T08:16:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 92/166 2023-11-08T08:16:53.743 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 93/166 2023-11-08T08:16:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 94/166 2023-11-08T08:16:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 95/166 2023-11-08T08:16:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.7-17.el8.x86_64 96/166 2023-11-08T08:16:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 97/166 2023-11-08T08:16:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-43.el8.x86_64 98/166 2023-11-08T08:16:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 98/166 2023-11-08T08:16:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 99/166 2023-11-08T08:16:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 100/166 2023-11-08T08:16:57.669 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 101/166 2023-11-08T08:16:58.105 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 101/166 2023-11-08T08:16:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 101/166 2023-11-08T08:16:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 102/166 2023-11-08T08:16:59.095 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 102/166 2023-11-08T08:16:59.321 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 103/166 2023-11-08T08:17:17.928 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 103/166 2023-11-08T08:17:17.929 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2023-11-08T08:17:17.929 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2023-11-08T08:17:17.929 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /dev 2023-11-08T08:17:17.929 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /run 2023-11-08T08:17:17.929 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2023-11-08T08:17:17.930 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2023-11-08T08:17:17.930 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2023-11-08T08:17:17.930 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2023-11-08T08:17:17.930 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /dev 2023-11-08T08:17:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:18.293 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 104/166 2023-11-08T08:17:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 104/166 2023-11-08T08:17:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 105/166 2023-11-08T08:17:20.281 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 105/166 2023-11-08T08:17:20.510 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 106/166 2023-11-08T08:17:20.958 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 106/166 2023-11-08T08:17:21.168 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 107/166 2023-11-08T08:17:21.191 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 107/166 2023-11-08T08:17:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 108/166 2023-11-08T08:17:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 109/166 2023-11-08T08:17:21.776 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 110/166 2023-11-08T08:17:21.858 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 110/166 2023-11-08T08:17:22.135 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-8.el8.x86_64 111/166 2023-11-08T08:17:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.1-8.el8.noarch 112/166 2023-11-08T08:17:22.885 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.0.0-2698.gb820560f.el8.noarch 113/166 2023-11-08T08:17:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:18.0.0-2698.gb820560f.el8.noarch 113/166 2023-11-08T08:17:23.182 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2698.gb820560f.e 114/166 2023-11-08T08:17:23.398 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2698.gb820560f. 115/166 2023-11-08T08:17:23.609 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noa 116/166 2023-11-08T08:17:23.819 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noa 116/166 2023-11-08T08:17:24.038 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.n 117/166 2023-11-08T08:17:24.999 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.n 117/166 2023-11-08T08:17:25.225 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2698.gb82 118/166 2023-11-08T08:17:25.314 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2698.gb82 118/166 2023-11-08T08:17:25.687 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2698.gb820560f.el 119/166 2023-11-08T08:17:25.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 120/166 2023-11-08T08:17:25.940 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 120/166 2023-11-08T08:17:26.084 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 121/166 2023-11-08T08:17:26.094 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 121/166 2023-11-08T08:17:26.315 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:18.0.0-2698.gb820560f.el8.x86_64 122/166 2023-11-08T08:17:26.458 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-25.el8.x86_64 123/166 2023-11-08T08:17:30.856 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 123/166 2023-11-08T08:17:32.301 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:18.0.0-2698.gb820560f.el8.x86_64 124/166 2023-11-08T08:17:32.488 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 125/166 2023-11-08T08:17:32.521 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 125/166 2023-11-08T08:17:32.716 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2698.gb8205 126/166 2023-11-08T08:17:32.758 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2698.gb8205 126/166 2023-11-08T08:17:32.958 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 127/166 2023-11-08T08:17:33.301 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 127/166 2023-11-08T08:17:33.500 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 128/166 2023-11-08T08:17:33.712 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 128/166 2023-11-08T08:17:34.047 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 129/166 2023-11-08T08:17:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 130/166 2023-11-08T08:17:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:1.1.1k-9.el8.x86_64 131/166 2023-11-08T08:17:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 132/166 2023-11-08T08:17:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/166 2023-11-08T08:17:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/166 2023-11-08T08:17:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:18.0.0-2698.gb820560f.el8.x86_ 135/166 2023-11-08T08:17:36.219 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 136/166 2023-11-08T08:17:36.400 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 137/166 2023-11-08T08:17:37.409 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 138/166 2023-11-08T08:17:37.674 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 138/166 2023-11-08T08:17:37.736 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 138/166 2023-11-08T08:17:37.995 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 139/166 2023-11-08T08:17:38.199 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:18.0.0-2698.gb820560f.el8.noarch 140/166 2023-11-08T08:17:38.473 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 141/166 2023-11-08T08:17:38.801 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:18.0.0-2698.gb820560f.el8.x86_64 142/166 2023-11-08T08:17:39.092 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 143/166 2023-11-08T08:17:39.232 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-11.e 144/166 2023-11-08T08:17:39.249 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-11.e 144/166 2023-11-08T08:17:39.437 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 145/166 2023-11-08T08:17:39.681 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.1-9.el8.x86_64 146/166 2023-11-08T08:17:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 146/166 2023-11-08T08:17:39.998 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.0.4-10.el8.x86_64 147/166 2023-11-08T08:17:40.011 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 147/166 2023-11-08T08:17:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 148/166 2023-11-08T08:17:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 149/166 2023-11-08T08:17:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/166 2023-11-08T08:17:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/166 2023-11-08T08:17:40.768 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/166 2023-11-08T08:17:40.768 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/166 2023-11-08T08:17:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/166 2023-11-08T08:17:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 153/166 2023-11-08T08:17:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 154/166 2023-11-08T08:17:41.193 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 154/166 2023-11-08T08:17:41.194 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 155/166 2023-11-08T08:17:41.459 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 155/166 2023-11-08T08:17:41.628 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 156/166 2023-11-08T08:17:41.850 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 157/166 2023-11-08T08:17:42.021 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 158/166 2023-11-08T08:17:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 159/166 2023-11-08T08:17:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 160/166 2023-11-08T08:17:42.475 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 160/166 2023-11-08T08:17:42.476 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 161/166 2023-11-08T08:17:42.668 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 161/166 2023-11-08T08:17:42.668 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 162/166 2023-11-08T08:17:42.851 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 162/166 2023-11-08T08:17:42.851 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 163/166 2023-11-08T08:17:43.059 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 163/166 2023-11-08T08:17:43.059 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 164/166 2023-11-08T08:17:43.259 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 164/166 2023-11-08T08:17:43.416 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 165/166 2023-11-08T08:17:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 166/166 2023-11-08T08:17:44.158 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libacl-2.2.53-1.el8.x86_64 166/166 2023-11-08T08:17:44.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:18.0.0-2698.gb820560f.el8.x86_64 1/166 2023-11-08T08:17:44.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 2/166 2023-11-08T08:17:44.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 3/166 2023-11-08T08:17:44.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 4/166 2023-11-08T08:17:44.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2698.gb8205 5/166 2023-11-08T08:17:44.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 6/166 2023-11-08T08:17:44.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 7/166 2023-11-08T08:17:44.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 8/166 2023-11-08T08:17:44.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 9/166 2023-11-08T08:17:44.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 10/166 2023-11-08T08:17:44.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 11/166 2023-11-08T08:17:44.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:18.0.0-2698.gb820560f.el8.x86_64 12/166 2023-11-08T08:17:44.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 13/166 2023-11-08T08:17:44.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:18.0.0-2698.gb820560f.el8.x86_ 14/166 2023-11-08T08:17:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 15/166 2023-11-08T08:17:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 16/166 2023-11-08T08:17:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:18.0.0-2698.gb820560f.el8.x86_6 17/166 2023-11-08T08:17:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86 18/166 2023-11-08T08:17:44.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 19/166 2023-11-08T08:17:44.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2698.gb820560f.el 20/166 2023-11-08T08:17:44.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:18.0.0-2698.gb820560f.el8. 21/166 2023-11-08T08:17:44.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:18.0.0-2698.gb820560f.el8.x86_6 22/166 2023-11-08T08:17:44.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:18.0.0-2698.gb820560f.el8.x86_64 23/166 2023-11-08T08:17:44.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:18.0.0-2698.gb820560f.el8.x86_64 24/166 2023-11-08T08:17:44.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:18.0.0-2698.gb820560f.el8.x86_64 25/166 2023-11-08T08:17:44.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 26/166 2023-11-08T08:17:44.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 27/166 2023-11-08T08:17:44.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:18.0.0-2698.gb820560f.el8.x86_64 28/166 2023-11-08T08:17:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2698.gb820560f. 29/166 2023-11-08T08:17:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noa 30/166 2023-11-08T08:17:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.n 31/166 2023-11-08T08:17:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2698.gb82 32/166 2023-11-08T08:17:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2698.gb820560f.el 33/166 2023-11-08T08:17:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 34/166 2023-11-08T08:17:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2698.gb820560f.e 35/166 2023-11-08T08:17:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 36/166 2023-11-08T08:17:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:18.0.0-2698.gb820560f.el8.noarch 37/166 2023-11-08T08:17:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:18.0.0-2698.gb820560f.el8.noarch 38/166 2023-11-08T08:17:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/166 2023-11-08T08:17:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/166 2023-11-08T08:17:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/166 2023-11-08T08:17:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-8.el8.x86_64 42/166 2023-11-08T08:17:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/166 2023-11-08T08:17:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 44/166 2023-11-08T08:17:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/166 2023-11-08T08:17:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/166 2023-11-08T08:17:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 47/166 2023-11-08T08:17:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/166 2023-11-08T08:17:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/166 2023-11-08T08:17:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/166 2023-11-08T08:17:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/166 2023-11-08T08:17:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/166 2023-11-08T08:17:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/166 2023-11-08T08:17:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 54/166 2023-11-08T08:17:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/166 2023-11-08T08:17:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/166 2023-11-08T08:17:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/166 2023-11-08T08:17:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/166 2023-11-08T08:17:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/166 2023-11-08T08:17:44.171 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/166 2023-11-08T08:17:44.171 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/166 2023-11-08T08:17:44.171 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/166 2023-11-08T08:17:44.171 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/166 2023-11-08T08:17:44.171 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/166 2023-11-08T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/166 2023-11-08T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 66/166 2023-11-08T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/166 2023-11-08T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/166 2023-11-08T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/166 2023-11-08T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/166 2023-11-08T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 71/166 2023-11-08T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 72/166 2023-11-08T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/166 2023-11-08T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/166 2023-11-08T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/166 2023-11-08T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/166 2023-11-08T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/166 2023-11-08T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/166 2023-11-08T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 79/166 2023-11-08T08:17:44.175 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/166 2023-11-08T08:17:44.175 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/166 2023-11-08T08:17:44.175 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 82/166 2023-11-08T08:17:44.175 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/166 2023-11-08T08:17:44.176 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/166 2023-11-08T08:17:44.176 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:1.1.1k-9.el8.x86_64 85/166 2023-11-08T08:17:44.176 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/166 2023-11-08T08:17:44.176 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/166 2023-11-08T08:17:44.176 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/166 2023-11-08T08:17:44.176 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/166 2023-11-08T08:17:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 90/166 2023-11-08T08:17:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/166 2023-11-08T08:17:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/166 2023-11-08T08:17:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/166 2023-11-08T08:17:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/166 2023-11-08T08:17:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/166 2023-11-08T08:17:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/166 2023-11-08T08:17:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 97/166 2023-11-08T08:17:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 98/166 2023-11-08T08:17:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 99/166 2023-11-08T08:17:44.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 100/166 2023-11-08T08:17:44.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 101/166 2023-11-08T08:17:44.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 102/166 2023-11-08T08:17:44.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 103/166 2023-11-08T08:17:44.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 104/166 2023-11-08T08:17:44.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 105/166 2023-11-08T08:17:44.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 106/166 2023-11-08T08:17:44.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 107/166 2023-11-08T08:17:44.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 108/166 2023-11-08T08:17:44.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 109/166 2023-11-08T08:17:44.181 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 110/166 2023-11-08T08:17:44.181 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 111/166 2023-11-08T08:17:44.181 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 112/166 2023-11-08T08:17:44.181 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 113/166 2023-11-08T08:17:44.181 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 114/166 2023-11-08T08:17:44.182 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-2.6-1.el8.noarch 115/166 2023-11-08T08:17:44.182 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 116/166 2023-11-08T08:17:44.182 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 117/166 2023-11-08T08:17:44.182 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 118/166 2023-11-08T08:17:44.182 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 119/166 2023-11-08T08:17:44.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 120/166 2023-11-08T08:17:44.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 121/166 2023-11-08T08:17:44.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 122/166 2023-11-08T08:17:44.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 123/166 2023-11-08T08:17:44.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 124/166 2023-11-08T08:17:44.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 125/166 2023-11-08T08:17:44.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 126/166 2023-11-08T08:17:44.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 127/166 2023-11-08T08:17:44.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 128/166 2023-11-08T08:17:44.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:18.0.0-2698.gb820560f.el8.x86_64 129/166 2023-11-08T08:17:44.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/166 2023-11-08T08:17:44.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 131/166 2023-11-08T08:17:44.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/166 2023-11-08T08:17:44.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-11.e 133/166 2023-11-08T08:17:44.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 134/166 2023-11-08T08:17:44.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.2.53-3.el8.x86_64 135/166 2023-11-08T08:17:44.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.2.53-1.el8.x86_64 136/166 2023-11-08T08:17:44.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el8.x86_64 137/166 2023-11-08T08:17:44.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 138/166 2023-11-08T08:17:44.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 139/166 2023-11-08T08:17:44.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 140/166 2023-11-08T08:17:44.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 141/166 2023-11-08T08:17:44.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 142/166 2023-11-08T08:17:44.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 143/166 2023-11-08T08:17:44.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 144/166 2023-11-08T08:17:44.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 145/166 2023-11-08T08:17:44.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 146/166 2023-11-08T08:17:44.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 147/166 2023-11-08T08:17:44.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 148/166 2023-11-08T08:17:44.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 149/166 2023-11-08T08:17:44.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 150/166 2023-11-08T08:17:44.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 151/166 2023-11-08T08:17:44.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 152/166 2023-11-08T08:17:44.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 153/166 2023-11-08T08:17:44.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 154/166 2023-11-08T08:17:44.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 155/166 2023-11-08T08:17:44.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 156/166 2023-11-08T08:17:44.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 157/166 2023-11-08T08:17:44.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 158/166 2023-11-08T08:17:44.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 159/166 2023-11-08T08:17:44.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 160/166 2023-11-08T08:17:44.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 161/166 2023-11-08T08:17:44.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 162/166 2023-11-08T08:17:44.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 163/166 2023-11-08T08:17:44.192 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 164/166 2023-11-08T08:17:44.192 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 165/166 2023-11-08T08:17:46.416 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 166/166 2023-11-08T08:17:46.417 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:46.417 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2023-11-08T08:17:46.417 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-initscripts-updown-1:1.40.16-11.el8.noarch 2023-11-08T08:17:46.417 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.2.53-3.el8.x86_64 2023-11-08T08:17:46.417 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.189-3.el8.x86_64 2023-11-08T08:17:46.418 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.189-3.el8.x86_64 2023-11-08T08:17:46.418 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.189-3.el8.x86_64 2023-11-08T08:17:46.418 INFO:teuthology.orchestra.run.smithi116.stdout: iproute-6.2.0-5.el8.x86_64 2023-11-08T08:17:46.418 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.2.53-3.el8.x86_64 2023-11-08T08:17:46.418 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.2.53-3.el8.x86_64 2023-11-08T08:17:46.419 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.419 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.419 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.419 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.419 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.420 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.420 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.420 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.420 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.420 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.8-7.el8.x86_64 2023-11-08T08:17:46.421 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-5.0.0-12.el8.x86_64 2023-11-08T08:17:46.421 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-08T08:17:46.421 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.1-8.el8.noarch 2023-11-08T08:17:46.421 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.0.4-10.el8.x86_64 2023-11-08T08:17:46.422 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.422 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.422 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.422 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.422 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.424 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.424 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.424 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.424 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.424 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.425 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.425 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.425 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.425 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.425 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.426 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.426 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:18.0.0-2698.gb820560f.el8.noarch 2023-11-08T08:17:46.426 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2023-11-08T08:17:46.426 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.1-9.el8.x86_64 2023-11-08T08:17:46.426 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.7-17.el8.x86_64 2023-11-08T08:17:46.427 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-08T08:17:46.427 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-8.el8.x86_64 2023-11-08T08:17:46.427 INFO:teuthology.orchestra.run.smithi116.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-08T08:17:46.427 INFO:teuthology.orchestra.run.smithi116.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-11-08T08:17:46.427 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-08T08:17:46.428 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.428 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.428 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.428 INFO:teuthology.orchestra.run.smithi116.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-11-08T08:17:46.428 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-08T08:17:46.429 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-08T08:17:46.429 INFO:teuthology.orchestra.run.smithi116.stdout: libpq-13.5-1.el8.x86_64 2023-11-08T08:17:46.429 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-08T08:17:46.429 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-08T08:17:46.429 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.430 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.430 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-08T08:17:46.430 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.430 INFO:teuthology.orchestra.run.smithi116.stdout: libselinux-devel-2.9-8.el8.x86_64 2023-11-08T08:17:46.430 INFO:teuthology.orchestra.run.smithi116.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-08T08:17:46.431 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-08T08:17:46.431 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-08T08:17:46.431 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2023-11-08T08:17:46.431 INFO:teuthology.orchestra.run.smithi116.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-11-08T08:17:46.431 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2023-11-08T08:17:46.432 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-08T08:17:46.432 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-08T08:17:46.432 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-08T08:17:46.432 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-08T08:17:46.432 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:1.1.1k-9.el8.x86_64 2023-11-08T08:17:46.433 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-11-08T08:17:46.433 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-11-08T08:17:46.433 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-11-08T08:17:46.433 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2023-11-08T08:17:46.433 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2023-11-08T08:17:46.434 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2023-11-08T08:17:46.434 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-08T08:17:46.434 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-08T08:17:46.434 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-08T08:17:46.434 INFO:teuthology.orchestra.run.smithi116.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-08T08:17:46.435 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-08T08:17:46.435 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.435 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.435 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.436 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-08T08:17:46.436 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-08T08:17:46.436 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-08T08:17:46.436 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-08T08:17:46.436 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-08T08:17:46.437 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-08T08:17:46.437 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-08T08:17:46.437 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-08T08:17:46.437 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-08T08:17:46.437 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-08T08:17:46.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-08T08:17:46.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-08T08:17:46.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-08T08:17:46.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-08T08:17:46.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-08T08:17:46.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-08T08:17:46.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-08T08:17:46.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-08T08:17:46.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-08T08:17:46.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-08T08:17:46.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-08T08:17:46.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-08T08:17:46.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-2.6-1.el8.noarch 2023-11-08T08:17:46.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-08T08:17:46.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-08T08:17:46.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-08T08:17:46.441 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-08T08:17:46.441 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-08T08:17:46.441 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-08T08:17:46.441 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.441 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.441 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-08T08:17:46.442 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-08T08:17:46.442 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.442 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-08T08:17:46.442 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-08T08:17:46.442 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-08T08:17:46.443 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-08T08:17:46.443 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-08T08:17:46.443 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-08T08:17:46.443 INFO:teuthology.orchestra.run.smithi116.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-08T08:17:46.443 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-08T08:17:46.444 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-08T08:17:46.444 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-08T08:17:46.444 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-08T08:17:46.444 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-08T08:17:46.444 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-08T08:17:46.445 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.445 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.445 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:18.0.0-2698.gb820560f.el8.x86_64 2023-11-08T08:17:46.445 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-08T08:17:46.445 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-08T08:17:46.446 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-08T08:17:46.446 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2023-11-08T08:17:46.446 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-08T08:17:46.447 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-11-08T08:17:46.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:46.447 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-08T08:17:46.614 DEBUG:teuthology.parallel:result is None 2023-11-08T08:17:46.615 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b820560f30d310ab2570761b9a294523225d6998 2023-11-08T08:17:46.838 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-08T08:17:46.862 INFO:teuthology.orchestra.run.smithi116.stdout:18.0.0-2698.gb820560f.el8 2023-11-08T08:17:46.863 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2698.gb820560f.el8 2023-11-08T08:17:46.863 INFO:teuthology.task.install:The correct ceph version 18.0.0-2698.gb820560f is installed. 2023-11-08T08:17:46.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b820560f30d310ab2570761b9a294523225d6998 2023-11-08T08:17:47.088 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-08T08:17:47.116 INFO:teuthology.orchestra.run.smithi129.stdout:18.0.0-2698.gb820560f.el8 2023-11-08T08:17:47.116 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2698.gb820560f.el8 2023-11-08T08:17:47.117 INFO:teuthology.task.install:The correct ceph version 18.0.0-2698.gb820560f is installed. 2023-11-08T08:17:47.117 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-08T08:17:47.118 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:47.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-08T08:17:47.150 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:47.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-08T08:17:47.190 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-08T08:17:47.191 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:47.192 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2023-11-08T08:17:47.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-08T08:17:47.283 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:47.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2023-11-08T08:17:47.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-08T08:17:47.385 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-08T08:17:47.385 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:47.386 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-08T08:17:47.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-08T08:17:47.480 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:47.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-08T08:17:47.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-08T08:17:47.575 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-08T08:17:47.576 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:47.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2023-11-08T08:17:47.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-08T08:17:47.670 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:47.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2023-11-08T08:17:47.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-08T08:17:47.766 INFO:teuthology.run_tasks:Running task ceph... 2023-11-08T08:17:47.832 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-08T08:17:47.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2023-11-08T08:17:47.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2023-11-08T08:17:47.863 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-08T08:17:47.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-08T08:17:47.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-08T08:17:47.929 INFO:tasks.ceph:Creating extra log directories... 2023-11-08T08:17:47.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-08T08:17:47.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-08T08:17:48.004 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-08T08:17:48.005 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-08T08:17:48.005 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-08T08:17:48.006 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:48.006 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-08T08:17:48.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-08T08:17:48.164 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-08T08:17:48.192 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:48.192 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-08T08:17:48.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-08T08:17:48.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-08T08:17:48.354 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-08T08:17:48.354 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-08T08:17:48.355 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824', 'branch': 'wip-neesingh-testing-54557', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}', 'email': 'neesingh@redhat.com', 'first_in_suite': False, 'job_id': '7451824', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-neesingh-testing-54557'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'b820560f30d310ab2570761b9a294523225d6998'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b820560f30d310ab2570761b9a294523225d6998'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-neesingh-testing-54557', 'sha1': 'b820560f30d310ab2570761b9a294523225d6998'}}, 'owner': 'scheduled_neesingh@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5135, 'sha1': 'b820560f30d310ab2570761b9a294523225d6998', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-neesingh-testing-54557', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b820560f30d310ab2570761b9a294523225d6998', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmWrVyaQLvu7kH/xOHecmPZs0Wv4Vnk8ebDAlklT4ioAMKeecUfXKCBKWoONiWEWSMBVYZ7doSnrlZcsf0LePE=', 'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP75L6TR03Z2dV6ScTobCTEsiEwEJDOeIRunLkcy5G1avJ+xdQ9cy9Bds1buAVhxI7AAzWZJ//guXa7HXaHSMi0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b820560f30d310ab2570761b9a294523225d6998'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-08_07:51:45', 'tube': 'smithi', 'user': 'neesingh', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714'} 2023-11-08T08:17:48.356 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-08T08:17:48.358 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-08T08:17:48.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T08:17:48.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T08:17:48.473 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:48.474 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2023-11-08T08:17:48.498 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-08T08:17:48.499 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2023-11-08T08:17:48.556 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-08T08:17:48.557 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:48.557 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 396502 Links: 1 2023-11-08T08:17:48.557 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:48.557 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:48.558 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-08 08:17:43.789118150 +0000 2023-11-08T08:17:48.558 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-08 08:10:29.827669469 +0000 2023-11-08T08:17:48.558 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-08 08:10:29.827669469 +0000 2023-11-08T08:17:48.558 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-11-08T08:17:48.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-08T08:17:48.622 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-08T08:17:48.622 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-08T08:17:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00021902 s, 2.3 MB/s 2023-11-08T08:17:48.624 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-08T08:17:48.682 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2023-11-08T08:17:48.741 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-08T08:17:48.741 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:48.741 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 397674 Links: 1 2023-11-08T08:17:48.741 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:48.742 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:48.742 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-08 08:17:43.789118150 +0000 2023-11-08T08:17:48.742 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-08 08:10:30.345661029 +0000 2023-11-08T08:17:48.742 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-08 08:10:30.345661029 +0000 2023-11-08T08:17:48.742 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-11-08T08:17:48.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-08T08:17:48.806 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-08T08:17:48.806 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-08T08:17:48.806 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000185976 s, 2.8 MB/s 2023-11-08T08:17:48.807 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-08T08:17:48.867 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2023-11-08T08:17:48.925 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-08T08:17:48.926 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:48.926 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 392892 Links: 1 2023-11-08T08:17:48.926 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:48.926 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:48.926 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-08 08:17:43.789118150 +0000 2023-11-08T08:17:48.927 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-08 08:10:30.865652557 +0000 2023-11-08T08:17:48.927 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-08 08:10:30.865652557 +0000 2023-11-08T08:17:48.927 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-11-08T08:17:48.927 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-08T08:17:48.990 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-08T08:17:48.991 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-08T08:17:48.991 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000189743 s, 2.7 MB/s 2023-11-08T08:17:48.992 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-08T08:17:49.051 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2023-11-08T08:17:49.110 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-08T08:17:49.111 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:49.111 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 399756 Links: 1 2023-11-08T08:17:49.111 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:49.111 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:49.111 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-08 08:17:43.789118150 +0000 2023-11-08T08:17:49.112 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-08 08:10:31.377644215 +0000 2023-11-08T08:17:49.112 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-08 08:10:31.377644215 +0000 2023-11-08T08:17:49.112 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-11-08T08:17:49.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-08T08:17:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-08T08:17:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-08T08:17:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000238474 s, 2.1 MB/s 2023-11-08T08:17:49.179 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-08T08:17:49.237 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-08T08:17:49.238 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:49.238 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2023-11-08T08:17:49.258 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-08T08:17:49.258 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2023-11-08T08:17:49.317 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-08T08:17:49.317 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:49.317 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 398479 Links: 1 2023-11-08T08:17:49.317 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:49.318 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:49.318 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-11-08 08:13:55.731590962 +0000 2023-11-08T08:17:49.318 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-11-08 08:10:29.817694441 +0000 2023-11-08T08:17:49.318 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-11-08 08:10:29.817694441 +0000 2023-11-08T08:17:49.319 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-11-08T08:17:49.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-08T08:17:49.384 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-11-08T08:17:49.384 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-11-08T08:17:49.384 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000240275 s, 2.1 MB/s 2023-11-08T08:17:49.385 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-08T08:17:49.445 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2023-11-08T08:17:49.504 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-08T08:17:49.504 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:49.504 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 390449 Links: 1 2023-11-08T08:17:49.505 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:49.505 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:49.505 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-11-08 08:13:55.731590962 +0000 2023-11-08T08:17:49.505 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-11-08 08:10:30.357683043 +0000 2023-11-08T08:17:49.505 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-11-08 08:10:30.357683043 +0000 2023-11-08T08:17:49.506 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-11-08T08:17:49.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-08T08:17:49.570 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-11-08T08:17:49.570 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-11-08T08:17:49.570 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000260265 s, 2.0 MB/s 2023-11-08T08:17:49.571 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-08T08:17:49.630 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2023-11-08T08:17:49.688 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-08T08:17:49.688 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:49.689 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 390528 Links: 1 2023-11-08T08:17:49.689 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:49.689 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:49.689 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-11-08 08:13:55.731590962 +0000 2023-11-08T08:17:49.689 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-11-08 08:10:30.865672321 +0000 2023-11-08T08:17:49.690 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-11-08 08:10:30.865672321 +0000 2023-11-08T08:17:49.690 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-11-08T08:17:49.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-08T08:17:49.753 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-11-08T08:17:49.754 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-11-08T08:17:49.754 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000235198 s, 2.2 MB/s 2023-11-08T08:17:49.755 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-08T08:17:49.813 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2023-11-08T08:17:49.871 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-08T08:17:49.871 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T08:17:49.871 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 390623 Links: 1 2023-11-08T08:17:49.872 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T08:17:49.872 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T08:17:49.872 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2023-11-08 08:13:55.731590962 +0000 2023-11-08T08:17:49.872 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2023-11-08 08:10:31.375661557 +0000 2023-11-08T08:17:49.872 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2023-11-08 08:10:31.375661557 +0000 2023-11-08T08:17:49.873 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2023-11-08T08:17:49.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-08T08:17:49.936 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2023-11-08T08:17:49.936 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2023-11-08T08:17:49.936 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000187006 s, 2.7 MB/s 2023-11-08T08:17:49.937 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-08T08:17:49.996 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-08T08:17:49.996 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-08T08:17:49.997 INFO:tasks.ceph:Generating config... 2023-11-08T08:17:49.998 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-08T08:17:49.999 INFO:tasks.ceph:[client] debug client = 20 2023-11-08T08:17:49.999 INFO:tasks.ceph:[client] debug ms = 1 2023-11-08T08:17:49.999 INFO:tasks.ceph:[client] pid file = 2023-11-08T08:17:49.999 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-08T08:17:50.000 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-08T08:17:50.000 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-08T08:17:50.000 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-08T08:17:50.000 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-08T08:17:50.001 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-08T08:17:50.001 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-08T08:17:50.001 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-08T08:17:50.001 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-08T08:17:50.002 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-08T08:17:50.002 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-08T08:17:50.002 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-08T08:17:50.003 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-08T08:17:50.003 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-08T08:17:50.004 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-08T08:17:50.004 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-08T08:17:50.004 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-08T08:17:50.004 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-08T08:17:50.005 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-08T08:17:50.005 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-08T08:17:50.005 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-08T08:17:50.005 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-08T08:17:50.006 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-08T08:17:50.006 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-08T08:17:50.006 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-08T08:17:50.007 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-08T08:17:50.007 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-08T08:17:50.007 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-08T08:17:50.007 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-08T08:17:50.008 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-08T08:17:50.008 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-08T08:17:50.008 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-08T08:17:50.008 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-08T08:17:50.009 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-08T08:17:50.009 INFO:tasks.ceph:Setting up mon.a... 2023-11-08T08:17:50.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-08T08:17:50.058 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.keyring 2023-11-08T08:17:50.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-08T08:17:50.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T08:17:50.133 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.116'), ('mon.b', '172.21.15.129'), ('mon.c', '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]')] 2023-11-08T08:17:50.134 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116,172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-08T08:17:50.135 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:50.135 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T08:17:50.191 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --add b 172.21.15.129 --addv c '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:17:50.280 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:17:50.280 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid 2f470591-d661-4b50-b7a5-062dfcf46b1d 2023-11-08T08:17:50.280 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = pacific 2023-11-08T08:17:50.280 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2023-11-08T08:17:50.281 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 2f470591-d661-4b50-b7a5-062dfcf46b1d 2023-11-08T08:17:50.281 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2023-11-08T08:17:50.280099+0000 2023-11-08T08:17:50.281 INFO:teuthology.orchestra.run.smithi116.stdout:created 2023-11-08T08:17:50.280099+0000 2023-11-08T08:17:50.281 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 16 (pacific) 2023-11-08T08:17:50.281 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2023-11-08T08:17:50.282 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-11-08T08:17:50.282 INFO:teuthology.orchestra.run.smithi116.stdout:1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2023-11-08T08:17:50.282 INFO:teuthology.orchestra.run.smithi116.stdout:2: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.c 2023-11-08T08:17:50.282 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-08T08:17:50.283 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T08:17:50.301 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2f470591-d661-4b50-b7a5-062dfcf46b1d... 2023-11-08T08:17:50.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-08T08:17:50.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-08T08:17:50.385 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2023-11-08T08:17:50.385 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2023-11-08T08:17:50.386 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T08:17:50.386 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2023-11-08T08:17:50.386 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.386 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2023-11-08T08:17:50.387 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.387 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2023-11-08T08:17:50.387 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.387 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2023-11-08T08:17:50.388 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2023-11-08T08:17:50.388 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.388 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2023-11-08T08:17:50.388 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2023-11-08T08:17:50.388 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.388 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T08:17:50.389 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2023-11-08T08:17:50.389 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.389 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T08:17:50.389 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T08:17:50.389 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.390 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2023-11-08T08:17:50.390 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.390 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2023-11-08T08:17:50.390 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2023-11-08T08:17:50.390 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2023-11-08T08:17:50.391 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2023-11-08T08:17:50.391 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2023-11-08T08:17:50.391 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2023-11-08T08:17:50.391 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2023-11-08T08:17:50.391 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T08:17:50.391 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T08:17:50.392 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2023-11-08T08:17:50.392 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.392 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-08T08:17:50.392 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.392 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2023-11-08T08:17:50.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.393 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2023-11-08T08:17:50.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.393 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2023-11-08T08:17:50.394 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2023-11-08T08:17:50.394 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2023-11-08T08:17:50.394 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.116,172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790]" 2023-11-08T08:17:50.394 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 2f470591-d661-4b50-b7a5-062dfcf46b1d 2023-11-08T08:17:50.394 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.395 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2023-11-08T08:17:50.395 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2023-11-08T08:17:50.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.395 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2023-11-08T08:17:50.395 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2023-11-08T08:17:50.396 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.396 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2023-11-08T08:17:50.396 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2023-11-08T08:17:50.396 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.396 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2023-11-08T08:17:50.397 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2023-11-08T08:17:50.397 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2023-11-08T08:17:50.397 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.397 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2023-11-08T08:17:50.398 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2023-11-08T08:17:50.398 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.399 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2023-11-08T08:17:50.399 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.399 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2023-11-08T08:17:50.399 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.400 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2023-11-08T08:17:50.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.400 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2023-11-08T08:17:50.400 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2023-11-08T08:17:50.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.401 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2023-11-08T08:17:50.401 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2023-11-08T08:17:50.401 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2023-11-08T08:17:50.401 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2023-11-08T08:17:50.401 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2023-11-08T08:17:50.402 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2023-11-08T08:17:50.402 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2023-11-08T08:17:50.402 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2023-11-08T08:17:50.402 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2023-11-08T08:17:50.402 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-08T08:17:50.403 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2023-11-08T08:17:50.403 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2023-11-08T08:17:50.403 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T08:17:50.403 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2023-11-08T08:17:50.403 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2023-11-08T08:17:50.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2023-11-08T08:17:50.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2023-11-08T08:17:50.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2023-11-08T08:17:50.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.404 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2023-11-08T08:17:50.404 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-08T08:17:50.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2023-11-08T08:17:50.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-11-08T08:17:50.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-11-08T08:17:50.405 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-11-08T08:17:50.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-11-08T08:17:50.406 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2023-11-08T08:17:50.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.406 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2023-11-08T08:17:50.406 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-08T08:17:50.406 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-11-08T08:17:50.407 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2023-11-08T08:17:50.407 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-11-08T08:17:50.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2023-11-08T08:17:50.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2023-11-08T08:17:50.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-11-08T08:17:50.408 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2023-11-08T08:17:50.408 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2023-11-08T08:17:50.408 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-11-08T08:17:50.408 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.408 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T08:17:50.409 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T08:17:50.409 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T08:17:50.409 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.409 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T08:17:50.409 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T08:17:50.410 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T08:17:50.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.410 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2023-11-08T08:17:50.410 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2023-11-08T08:17:50.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.411 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2023-11-08T08:17:50.411 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2023-11-08T08:17:50.411 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:17:50.411 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2023-11-08T08:17:50.411 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2023-11-08T08:17:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2023-11-08T08:17:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2023-11-08T08:17:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T08:17:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T08:17:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2023-11-08T08:17:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2023-11-08T08:17:50.413 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-08T08:17:50.413 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2023-11-08T08:17:50.413 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2023-11-08T08:17:50.413 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2023-11-08T08:17:50.413 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2023-11-08T08:17:50.414 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2023-11-08T08:17:50.414 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2023-11-08T08:17:50.414 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2023-11-08T08:17:50.414 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2023-11-08T08:17:50.414 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2023-11-08T08:17:50.415 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2023-11-08T08:17:50.415 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2023-11-08T08:17:50.415 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2023-11-08T08:17:50.415 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2023-11-08T08:17:50.415 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-08T08:17:50.416 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2023-11-08T08:17:50.416 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2023-11-08T08:17:50.416 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2023-11-08T08:17:50.416 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2023-11-08T08:17:50.416 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2023-11-08T08:17:50.417 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2023-11-08T08:17:50.417 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2023-11-08T08:17:50.418 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2023-11-08T08:17:50.419 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2023-11-08T08:17:50.419 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T08:17:50.419 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2023-11-08T08:17:50.419 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.420 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2023-11-08T08:17:50.420 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.420 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2023-11-08T08:17:50.420 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.420 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2023-11-08T08:17:50.421 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2023-11-08T08:17:50.421 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.421 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2023-11-08T08:17:50.421 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2023-11-08T08:17:50.421 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.422 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T08:17:50.422 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2023-11-08T08:17:50.422 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.422 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T08:17:50.422 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T08:17:50.423 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.423 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2023-11-08T08:17:50.423 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.423 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2023-11-08T08:17:50.423 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2023-11-08T08:17:50.424 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2023-11-08T08:17:50.424 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2023-11-08T08:17:50.424 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2023-11-08T08:17:50.424 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2023-11-08T08:17:50.424 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2023-11-08T08:17:50.425 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T08:17:50.425 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T08:17:50.425 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2023-11-08T08:17:50.425 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.425 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-08T08:17:50.426 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.426 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2023-11-08T08:17:50.426 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.426 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2023-11-08T08:17:50.426 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.427 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2023-11-08T08:17:50.427 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2023-11-08T08:17:50.427 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2023-11-08T08:17:50.427 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.116,172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790]" 2023-11-08T08:17:50.427 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 2f470591-d661-4b50-b7a5-062dfcf46b1d 2023-11-08T08:17:50.428 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.428 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2023-11-08T08:17:50.428 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2023-11-08T08:17:50.428 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.429 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2023-11-08T08:17:50.429 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2023-11-08T08:17:50.429 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.429 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2023-11-08T08:17:50.429 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2023-11-08T08:17:50.430 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.430 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2023-11-08T08:17:50.430 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2023-11-08T08:17:50.430 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2023-11-08T08:17:50.430 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2023-11-08T08:17:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2023-11-08T08:17:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2023-11-08T08:17:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.432 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2023-11-08T08:17:50.432 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.432 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2023-11-08T08:17:50.432 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.432 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2023-11-08T08:17:50.433 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2023-11-08T08:17:50.433 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.433 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2023-11-08T08:17:50.433 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2023-11-08T08:17:50.433 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2023-11-08T08:17:50.434 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2023-11-08T08:17:50.434 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2023-11-08T08:17:50.434 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2023-11-08T08:17:50.434 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2023-11-08T08:17:50.434 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2023-11-08T08:17:50.435 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2023-11-08T08:17:50.435 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-11-08T08:17:50.437 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2023-11-08T08:17:50.438 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2023-11-08T08:17:50.438 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T08:17:50.438 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2023-11-08T08:17:50.438 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2023-11-08T08:17:50.438 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2023-11-08T08:17:50.439 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2023-11-08T08:17:50.439 INFO:teuthology.orchestra.run.smithi129.stdout: osd op complaint time = 180 2023-11-08T08:17:50.439 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.439 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2023-11-08T08:17:50.439 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-11-08T08:17:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2023-11-08T08:17:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2023-11-08T08:17:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2023-11-08T08:17:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2023-11-08T08:17:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2023-11-08T08:17:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2023-11-08T08:17:50.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.441 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2023-11-08T08:17:50.441 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-11-08T08:17:50.441 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2023-11-08T08:17:50.441 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2023-11-08T08:17:50.442 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2023-11-08T08:17:50.442 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2023-11-08T08:17:50.442 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2023-11-08T08:17:50.442 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2023-11-08T08:17:50.442 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2023-11-08T08:17:50.443 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2023-11-08T08:17:50.443 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2023-11-08T08:17:50.443 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.443 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T08:17:50.443 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T08:17:50.444 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T08:17:50.444 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.444 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T08:17:50.444 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T08:17:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T08:17:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2023-11-08T08:17:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2023-11-08T08:17:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2023-11-08T08:17:50.446 INFO:teuthology.orchestra.run.smithi129.stdout: mon op complaint time = 120 2023-11-08T08:17:50.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-11-08T08:17:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2023-11-08T08:17:50.446 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2023-11-08T08:17:50.446 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2023-11-08T08:17:50.447 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2023-11-08T08:17:50.447 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T08:17:50.447 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T08:17:50.447 INFO:teuthology.orchestra.run.smithi129.stdout: client mount timeout = 600 2023-11-08T08:17:50.447 INFO:teuthology.orchestra.run.smithi129.stdout: debug client = 20 2023-11-08T08:17:50.448 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-11-08T08:17:50.448 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2023-11-08T08:17:50.448 INFO:teuthology.orchestra.run.smithi129.stdout: rados mon op timeout = 900 2023-11-08T08:17:50.448 INFO:teuthology.orchestra.run.smithi129.stdout: rados osd op timeout = 900 2023-11-08T08:17:50.448 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2023-11-08T08:17:50.449 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2023-11-08T08:17:50.449 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2023-11-08T08:17:50.449 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2023-11-08T08:17:50.449 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2023-11-08T08:17:50.449 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2023-11-08T08:17:50.450 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2023-11-08T08:17:50.450 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2023-11-08T08:17:50.450 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds = 20 2023-11-08T08:17:50.450 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds balancer = 20 2023-11-08T08:17:50.450 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2023-11-08T08:17:50.451 INFO:teuthology.orchestra.run.smithi129.stdout: mds debug frag = True 2023-11-08T08:17:50.451 INFO:teuthology.orchestra.run.smithi129.stdout: mds debug scatterstat = True 2023-11-08T08:17:50.451 INFO:teuthology.orchestra.run.smithi129.stdout: mds op complaint time = 180 2023-11-08T08:17:50.451 INFO:teuthology.orchestra.run.smithi129.stdout: mds verify scatter = True 2023-11-08T08:17:50.451 INFO:teuthology.orchestra.run.smithi129.stdout: osd op complaint time = 180 2023-11-08T08:17:50.452 INFO:teuthology.orchestra.run.smithi129.stdout: rados mon op timeout = 900 2023-11-08T08:17:50.452 INFO:teuthology.orchestra.run.smithi129.stdout: rados osd op timeout = 900 2023-11-08T08:17:50.452 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-08T08:17:50.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-08T08:17:50.515 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-08T08:17:50.515 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:50.516 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-08T08:17:50.533 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:50.534 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-08T08:17:50.590 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:17:50.591 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:50.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-08T08:17:50.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T08:17:50.666 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:17:50.667 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:17:50.723 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:17:50.724 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:50.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-08T08:17:50.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T08:17:50.761 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:17:50.762 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:17:50.819 INFO:tasks.ceph:Setting up mon nodes... 2023-11-08T08:17:50.820 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-08T08:17:50.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-08T08:17:50.872 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-08T08:17:50.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-08T08:17:50.972 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-08T08:17:50.974 INFO:tasks.ceph:Setting up mds nodes... 2023-11-08T08:17:50.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-08T08:17:51.028 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-08T08:17:51.030 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-08T08:17:51.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-08T08:17:51.150 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-08T08:17:51.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-08T08:17:51.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-08T08:17:51.233 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-08T08:17:51.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-08T08:17:51.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-08T08:17:51.354 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-08T08:17:51.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-08T08:17:51.383 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-08T08:17:51.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-08T08:17:51.427 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-08T08:17:51.439 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-08T08:17:51.522 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-08T08:17:51.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-08T08:17:51.578 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-08T08:17:51.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-08T08:17:51.634 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-08T08:17:51.645 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-08T08:17:51.646 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-08T08:17:51.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-08T08:17:51.674 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-08T08:17:51.674 INFO:tasks.ceph:role: osd.0 2023-11-08T08:17:51.674 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:17:51.675 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-08T08:17:51.748 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:17:51.748 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:17:51.748 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:17:51.748 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:17:51.749 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:17:51.749 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:17:51.750 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:17:51.751 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:17:51.751 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:17:51.751 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:17:52.173 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-08T08:17:52.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-08T08:17:52.175 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-08T08:17:52.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-08T08:17:52.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-08T08:17:52.463 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-08T08:17:52.464 INFO:tasks.ceph:role: osd.1 2023-11-08T08:17:52.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:17:52.464 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-08T08:17:52.537 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:17:52.538 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:17:52.538 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:17:52.538 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:17:52.538 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:17:52.539 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:17:52.539 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:17:52.539 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:17:52.539 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:17:52.539 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:17:52.962 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-08T08:17:52.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-08T08:17:52.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-08T08:17:53.075 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-08T08:17:53.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-08T08:17:53.175 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-08T08:17:53.176 INFO:tasks.ceph:role: osd.2 2023-11-08T08:17:53.176 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:17:53.176 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-08T08:17:53.250 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:17:53.250 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:17:53.250 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:17:53.251 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:17:53.251 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:17:53.251 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:17:53.251 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:17:53.251 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:17:53.252 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:17:53.252 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:17:53.699 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-08T08:17:53.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-08T08:17:53.701 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-08T08:17:53.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-08T08:17:53.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-08T08:17:53.979 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-08T08:17:53.980 INFO:tasks.ceph:role: osd.3 2023-11-08T08:17:53.980 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:17:53.980 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-08T08:17:54.055 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:17:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:17:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:17:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:17:54.056 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:17:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:17:54.056 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:17:54.057 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:17:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:17:54.057 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:17:54.493 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-08T08:17:54.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-08T08:17:54.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-08T08:17:54.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-08T08:17:54.717 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:17:54.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:54.820+0000 7fbeeb71d680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-08T08:17:54.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:54.820+0000 7fbeeb71d680 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-08T08:17:54.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:54.820+0000 7fbeeb71d680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-08T08:17:54.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:54.820+0000 7fbeeb71d680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-08T08:17:54.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:54.820+0000 7fbeeb71d680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-08T08:17:54.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:54.821+0000 7fbeeb71d680 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-08T08:17:57.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-08T08:17:57.706 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:17:57.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:57.795+0000 7febf8715680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-08T08:17:57.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:57.796+0000 7febf8715680 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-08T08:17:57.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:57.796+0000 7febf8715680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-08T08:17:57.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:57.796+0000 7febf8715680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-08T08:17:57.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:57.796+0000 7febf8715680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-08T08:17:57.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:17:57.796+0000 7febf8715680 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-08T08:18:00.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-08T08:18:00.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:00.780+0000 7f3395232680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-08T08:18:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:00.781+0000 7f3395232680 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-08T08:18:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:00.781+0000 7f3395232680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-08T08:18:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:00.781+0000 7f3395232680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-08T08:18:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:00.781+0000 7f3395232680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-08T08:18:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:00.781+0000 7f3395232680 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-08T08:18:03.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-08T08:18:03.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:03.751+0000 7f13ed24e680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-08T08:18:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:03.751+0000 7f13ed24e680 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-08T08:18:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:03.751+0000 7f13ed24e680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-08T08:18:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:03.751+0000 7f13ed24e680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-08T08:18:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:03.751+0000 7f13ed24e680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-08T08:18:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:03.751+0000 7f13ed24e680 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-08T08:18:06.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-08T08:18:06.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-08T08:18:06.664 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-08T08:18:06.665 INFO:tasks.ceph:role: osd.4 2023-11-08T08:18:06.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:18:06.665 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-08T08:18:06.742 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:18:06.743 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:18:06.743 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:18:06.743 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:18:06.743 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:18:06.744 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:18:06.744 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:18:06.744 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:18:06.744 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:18:06.744 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:18:07.171 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-11-08T08:18:07.173 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-11-08T08:18:07.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-08T08:18:07.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-08T08:18:07.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-08T08:18:07.457 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-08T08:18:07.457 INFO:tasks.ceph:role: osd.5 2023-11-08T08:18:07.458 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:18:07.458 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-08T08:18:07.533 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:18:07.533 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:18:07.533 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:18:07.533 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:18:07.534 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:18:07.534 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:18:07.534 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:18:07.534 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:18:07.535 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:18:07.535 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:18:07.954 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-11-08T08:18:07.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-11-08T08:18:07.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-08T08:18:08.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-08T08:18:08.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-08T08:18:08.175 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-08T08:18:08.176 INFO:tasks.ceph:role: osd.6 2023-11-08T08:18:08.176 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:18:08.176 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-08T08:18:08.253 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:18:08.253 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:18:08.254 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:18:08.254 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:18:08.254 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:18:08.254 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:18:08.254 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:18:08.255 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:18:08.255 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:18:08.255 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:18:08.695 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-11-08T08:18:08.697 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-11-08T08:18:08.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-08T08:18:08.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-08T08:18:08.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-08T08:18:08.967 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-08T08:18:08.968 INFO:tasks.ceph:role: osd.7 2023-11-08T08:18:08.968 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:18:08.968 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-08T08:18:09.046 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T08:18:09.046 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T08:18:09.046 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T08:18:09.046 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T08:18:09.047 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T08:18:09.047 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2023-11-08T08:18:09.047 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T08:18:09.047 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T08:18:09.047 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T08:18:09.048 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T08:18:09.488 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2023-11-08T08:18:09.490 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2023-11-08T08:18:09.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-08T08:18:09.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-08T08:18:09.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:09.803 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:09.802+0000 7fab0acd4680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-08T08:18:09.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:09.803+0000 7fab0acd4680 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-08T08:18:09.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:09.803+0000 7fab0acd4680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-08T08:18:09.804 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:09.803+0000 7fab0acd4680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-08T08:18:09.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:09.803+0000 7fab0acd4680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-08T08:18:09.805 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:09.803+0000 7fab0acd4680 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-08T08:18:12.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-08T08:18:12.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:12.781+0000 7fe14b203680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-08T08:18:12.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:12.782+0000 7fe14b203680 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-08T08:18:12.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:12.782+0000 7fe14b203680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-08T08:18:12.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:12.782+0000 7fe14b203680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-08T08:18:12.783 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:12.782+0000 7fe14b203680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-08T08:18:12.784 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:12.782+0000 7fe14b203680 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-08T08:18:15.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-08T08:18:15.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:15.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:15.758+0000 7fc717a1e680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-08T08:18:15.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:15.759+0000 7fc717a1e680 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-08T08:18:15.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:15.759+0000 7fc717a1e680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-08T08:18:15.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:15.759+0000 7fc717a1e680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-08T08:18:15.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:15.759+0000 7fc717a1e680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-08T08:18:15.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:15.759+0000 7fc717a1e680 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-08T08:18:18.357 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:18:18.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:18:18.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-08T08:18:18.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:18.722 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:18.721+0000 7fbc0da71680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-08T08:18:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:18.722+0000 7fbc0da71680 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-08T08:18:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:18.722+0000 7fbc0da71680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-08T08:18:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:18.722+0000 7fbc0da71680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-08T08:18:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:18.722+0000 7fbc0da71680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-08T08:18:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:18.722+0000 7fbc0da71680 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-08T08:18:21.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-08T08:18:21.609 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-08T08:18:21.609 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-08T08:18:21.642 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-08T08:18:21.714 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-08T08:18:21.784 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-08T08:18:21.852 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-08T08:18:21.922 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-08T08:18:21.989 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:21.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-08T08:18:22.056 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-08T08:18:22.089 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-08T08:18:22.159 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-08T08:18:22.229 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-08T08:18:22.297 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-08T08:18:22.366 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.367 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-08T08:18:22.435 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-08T08:18:22.502 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:22.503 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-08T08:18:22.522 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:22.522 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-08T08:18:22.580 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.581 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-08T08:18:22.599 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:22.600 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-08T08:18:22.657 INFO:tasks.ceph:Adding keys to all mons... 2023-11-08T08:18:22.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-08T08:18:22.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-08T08:18:22.700 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2023-11-08T08:18:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAuREtlXRIMNBAAC9TkKhc8QzTE+DK2eDs/bg== 2023-11-08T08:18:22.701 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2023-11-08T08:18:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtlwPmvARAAdJKHnQqKVPWnN7uQiHc+WQ== 2023-11-08T08:18:22.701 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2023-11-08T08:18:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtlNA7+CBAAkUCXKhY1qacysaUprxlU9w== 2023-11-08T08:18:22.701 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2023-11-08T08:18:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAyREtlwCT6MBAA0hUY6XxZE8Q/cW66704LAA== 2023-11-08T08:18:22.702 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2023-11-08T08:18:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA1REtlHxOBLxAAHR12A9jsXL7W8TeG55VeWg== 2023-11-08T08:18:22.702 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2023-11-08T08:18:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA4REtlcPidLhAAhNEoyldnbVRtP9IQr08wag== 2023-11-08T08:18:22.703 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2023-11-08T08:18:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA7REtlH/3XLBAASJ532Or0yrTC8qfr3kCziQ== 2023-11-08T08:18:22.703 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2023-11-08T08:18:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAuREtl1TH6ORAAQMEd7GQJFqk28L6HfaYPlg== 2023-11-08T08:18:22.703 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2023-11-08T08:18:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtlXUbqDRAATJnFxEiS//echgEFyiS+iw== 2023-11-08T08:18:22.704 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2023-11-08T08:18:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtltlgoFRAAwAkFfezamgYHcC9awLpLpA== 2023-11-08T08:18:22.704 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2023-11-08T08:18:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBBREtl5+beLxAAXM67eNV2/FzHU4+gp3zMng== 2023-11-08T08:18:22.705 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2023-11-08T08:18:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBEREtl4pChLhAA470kPHr4WFBGgwEi0CB95A== 2023-11-08T08:18:22.705 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2023-11-08T08:18:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBHREtlh5g/LRAAY7eRgyAdWK0ExY6Qaf6hgQ== 2023-11-08T08:18:22.705 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2023-11-08T08:18:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBKREtly78MKxAA5+d+0MekiMWQK1sAn1Ue6w== 2023-11-08T08:18:22.706 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-11-08T08:18:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtlHhh+GRAAKHMDRBv0YayhqERZiWXGAw== 2023-11-08T08:18:22.706 INFO:teuthology.orchestra.run.smithi116.stdout:[client.2] 2023-11-08T08:18:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtlKh8kHxAA/Ear8UA/wixquw0KBzet2A== 2023-11-08T08:18:22.707 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2023-11-08T08:18:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtl2ICCIhAAdgVi5BOXs1xqGLKgTLEXoQ== 2023-11-08T08:18:22.707 INFO:teuthology.orchestra.run.smithi116.stdout:[client.3] 2023-11-08T08:18:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvREtl17vQJRAA/R0BvGRHgw8ZySwL4KyuGg== 2023-11-08T08:18:22.722 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.y] 2023-11-08T08:18:22.722 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAuREtlXRIMNBAAC9TkKhc8QzTE+DK2eDs/bg== 2023-11-08T08:18:22.722 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2023-11-08T08:18:22.722 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtlwPmvARAAdJKHnQqKVPWnN7uQiHc+WQ== 2023-11-08T08:18:22.723 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2023-11-08T08:18:22.723 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtlNA7+CBAAkUCXKhY1qacysaUprxlU9w== 2023-11-08T08:18:22.723 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2023-11-08T08:18:22.723 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAyREtlwCT6MBAA0hUY6XxZE8Q/cW66704LAA== 2023-11-08T08:18:22.723 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2023-11-08T08:18:22.724 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA1REtlHxOBLxAAHR12A9jsXL7W8TeG55VeWg== 2023-11-08T08:18:22.724 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2023-11-08T08:18:22.724 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA4REtlcPidLhAAhNEoyldnbVRtP9IQr08wag== 2023-11-08T08:18:22.724 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2023-11-08T08:18:22.724 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQA7REtlH/3XLBAASJ532Or0yrTC8qfr3kCziQ== 2023-11-08T08:18:22.725 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2023-11-08T08:18:22.725 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAuREtl1TH6ORAAQMEd7GQJFqk28L6HfaYPlg== 2023-11-08T08:18:22.725 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2023-11-08T08:18:22.725 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtlXUbqDRAATJnFxEiS//echgEFyiS+iw== 2023-11-08T08:18:22.725 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2023-11-08T08:18:22.726 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtltlgoFRAAwAkFfezamgYHcC9awLpLpA== 2023-11-08T08:18:22.726 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2023-11-08T08:18:22.726 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBBREtl5+beLxAAXM67eNV2/FzHU4+gp3zMng== 2023-11-08T08:18:22.726 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2023-11-08T08:18:22.726 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBEREtl4pChLhAA470kPHr4WFBGgwEi0CB95A== 2023-11-08T08:18:22.727 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.6] 2023-11-08T08:18:22.727 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBHREtlh5g/LRAAY7eRgyAdWK0ExY6Qaf6hgQ== 2023-11-08T08:18:22.727 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.7] 2023-11-08T08:18:22.727 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBKREtly78MKxAA5+d+0MekiMWQK1sAn1Ue6w== 2023-11-08T08:18:22.727 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2023-11-08T08:18:22.728 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtlHhh+GRAAKHMDRBv0YayhqERZiWXGAw== 2023-11-08T08:18:22.728 INFO:teuthology.orchestra.run.smithi129.stdout:[client.2] 2023-11-08T08:18:22.728 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtlKh8kHxAA/Ear8UA/wixquw0KBzet2A== 2023-11-08T08:18:22.728 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2023-11-08T08:18:22.728 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtl2ICCIhAAdgVi5BOXs1xqGLKgTLEXoQ== 2023-11-08T08:18:22.728 INFO:teuthology.orchestra.run.smithi129.stdout:[client.3] 2023-11-08T08:18:22.729 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAvREtl17vQJRAA/R0BvGRHgw8ZySwL4KyuGg== 2023-11-08T08:18:22.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T08:18:22.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T08:18:22.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:22.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:22.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:22.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:22.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:22.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:22.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:22.996 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.049 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.161 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T08:18:23.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T08:18:23.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:23.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:23.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:23.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-08T08:18:23.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.336 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.501 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T08:18:23.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T08:18:23.789 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-08T08:18:23.790 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-08T08:18:23.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-08T08:18:23.915 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-08T08:18:23.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-08T08:18:23.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-08T08:18:24.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-08T08:18:24.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-08T08:18:24.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-08T08:18:24.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-08T08:18:24.287 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:24.290 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-08T08:18:24.344 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-08T08:18:24.345 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-08T08:18:24.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-08T08:18:24.348 INFO:tasks.ceph.mon.a:Started 2023-11-08T08:18:24.348 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-08T08:18:24.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-08T08:18:24.387 INFO:tasks.ceph.mon.b:Started 2023-11-08T08:18:24.387 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-08T08:18:24.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-08T08:18:24.390 INFO:tasks.ceph.mon.c:Started 2023-11-08T08:18:24.390 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-08T08:18:24.391 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-08T08:18:24.391 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-08T08:18:24.393 INFO:tasks.ceph.mgr.y:Started 2023-11-08T08:18:24.393 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-08T08:18:24.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-08T08:18:24.396 INFO:tasks.ceph.mgr.x:Started 2023-11-08T08:18:24.396 INFO:tasks.ceph:Setting crush tunables to default 2023-11-08T08:18:24.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd crush tunables default 2023-11-08T08:18:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.517+0000 7f3da1416700 1 Processor -- start 2023-11-08T08:18:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 -- start start 2023-11-08T08:18:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c06ac00 con 0x7f3d9c06a7b0 2023-11-08T08:18:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c15e0d0 con 0x7f3d9c0cdd50 2023-11-08T08:18:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3da1416700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0ce560 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3d9a7fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3d9a7fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40608/0 (socket says 172.21.15.116:40608) 2023-11-08T08:18:24.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3d9a7fc700 1 -- 172.21.15.116:0/3749435921 learned_addr learned my addr 172.21.15.116:0/3749435921 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:24.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3d9a7fc700 1 -- 172.21.15.116:0/3749435921 >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 msgr2=0x7f3d9c0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-08T08:18:24.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3d9a7fc700 1 -- 172.21.15.116:0/3749435921 >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 msgr2=0x7f3d9c0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-08T08:18:24.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.518+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-08T08:18:24.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-08T08:18:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9affd700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9affd700 1 -- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 msgr2=0x7f3d9c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-08T08:18:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9affd700 1 -- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 msgr2=0x7f3d9c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-08T08:18:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9affd700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-08T08:18:24.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9affd700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-08T08:18:24.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9b7fe700 1 -- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 msgr2=0x7f3d9c0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-08T08:18:24.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9b7fe700 1 -- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 msgr2=0x7f3d9c0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-08T08:18:24.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-08T08:18:24.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.519+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-08T08:18:24.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.116:3300/0 conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9affd700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 -- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 msgr2=0x7f3d9c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 -- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 msgr2=0x7f3d9c0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 -- 172.21.15.116:0/3749435921 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c15e320 con 0x7f3d9c06a7b0 2023-11-08T08:18:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0af5e0 tx=0x7f3d8c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc5aaa117777a8b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d99ffb700 1 -- 172.21.15.116:0/3749435921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c004230 con 0x7f3d9c06a7b0 2023-11-08T08:18:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d99ffb700 1 -- 172.21.15.116:0/3749435921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c004390 con 0x7f3d9c06a7b0 2023-11-08T08:18:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.720+0000 7f3d99ffb700 1 -- 172.21.15.116:0/3749435921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c0036a0 con 0x7f3d9c06a7b0 2023-11-08T08:18:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 -- 172.21.15.116:0/3749435921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c06a7b0 msgr2=0x7f3d9c0cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0af5e0 tx=0x7f3d8c009c30 comp rx=0 tx=0).stop 2023-11-08T08:18:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 -- 172.21.15.116:0/3749435921 shutdown_connections 2023-11-08T08:18:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 --2- 172.21.15.116:0/3749435921 >> v2:172.21.15.129:3300/0 conn(0x7f3d9c0cdd50 0x7f3d9c0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c15a4d0 0x7f3d9c06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 --2- 172.21.15.116:0/3749435921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c06a7b0 0x7f3d9c0cd810 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 -- 172.21.15.116:0/3749435921 >> 172.21.15.116:0/3749435921 conn(0x7f3d9c06c010 msgr2=0x7f3d9c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 -- 172.21.15.116:0/3749435921 shutdown_connections 2023-11-08T08:18:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.721+0000 7f3da1416700 1 -- 172.21.15.116:0/3749435921 wait complete. 2023-11-08T08:18:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.722+0000 7f3da1416700 1 Processor -- start 2023-11-08T08:18:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.722+0000 7f3da1416700 1 -- start start 2023-11-08T08:18:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3da1416700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c06a7b0 0x7f3d9c1f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3da1416700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9c0cdd50 0x7f3d9c0b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3da1416700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 0x7f3d9c0c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3da1416700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1e48b0 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3da1416700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1e4580 con 0x7f3d9c0cdd50 2023-11-08T08:18:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3da1416700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0b1190 con 0x7f3d9c06a7b0 2023-11-08T08:18:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 0x7f3d9c0c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9affd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c06a7b0 0x7f3d9c1f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 0x7f3d9c0c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40632/0 (socket says 172.21.15.116:40632) 2023-11-08T08:18:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 -- 172.21.15.116:0/1820204772 learned_addr learned my addr 172.21.15.116:0/1820204772 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9c0cdd50 0x7f3d9c0b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 -- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c06a7b0 msgr2=0x7f3d9c1f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c06a7b0 0x7f3d9c1f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 -- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9c0cdd50 msgr2=0x7f3d9c0b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9c0cdd50 0x7f3d9c0b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9b7fe700 1 -- 172.21.15.116:0/1820204772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c0095b0 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9a7fc700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9c0cdd50 0x7f3d9c0b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.723+0000 7f3d9affd700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c06a7b0 0x7f3d9c1f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3d9b7fe700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 0x7f3d9c0c9010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d940046e0 tx=0x7f3d940049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94007cc0 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9400f050 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c15e8e0 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94014650 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c15e600 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f3d94014aa0 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.724+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3d9401ea00 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.725+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c15a8a0 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.727+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f3d94007e20 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.818+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3d9c1e4a20 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.820+0000 7f3d837fe700 1 -- 172.21.15.116:0/1820204772 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3d9401e460 con 0x7f3d9c15a4d0 2023-11-08T08:18:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 msgr2=0x7f3d9c0c9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 0x7f3d9c0c9010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d940046e0 tx=0x7f3d940049f0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 shutdown_connections 2023-11-08T08:18:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d9c15a4d0 0x7f3d9c0c9010 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3d9c06a7b0 0x7f3d9c1f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 --2- 172.21.15.116:0/1820204772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9c0cdd50 0x7f3d9c0b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 >> 172.21.15.116:0/1820204772 conn(0x7f3d9c06c010 msgr2=0x7f3d9c14f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 shutdown_connections 2023-11-08T08:18:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:24.821+0000 7f3da1416700 1 -- 172.21.15.116:0/1820204772 wait complete. 2023-11-08T08:18:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2023-11-08T08:18:24.834 INFO:tasks.ceph:check_enable_crimson: False 2023-11-08T08:18:24.834 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-08T08:18:24.835 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:24.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-08T08:18:24.837 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:24.836+0000 7f0604c66240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-08T08:18:24.847 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:24.847+0000 7f8c088cb240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-08T08:18:24.861 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:24.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-08T08:18:24.927 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:24.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-08T08:18:25.004 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:25.003+0000 7f0604c66240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-08T08:18:25.007 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:18:25.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-08T08:18:25.022 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:25.021+0000 7f8c088cb240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-08T08:18:25.048 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:25.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-08T08:18:25.076 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:25.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-08T08:18:25.143 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:25.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-08T08:18:25.209 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:18:25.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-08T08:18:25.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 028d5009-e114-46ca-86e3-92451886a981 0 2023-11-08T08:18:25.356 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:25.355+0000 7f8c088cb240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-08T08:18:25.389 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:25.388+0000 7f0604c66240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-08T08:18:25.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.437+0000 7fa683d32700 1 Processor -- start 2023-11-08T08:18:25.439 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.438+0000 7fa683d32700 1 -- start start 2023-11-08T08:18:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.438+0000 7fa683d32700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa67c0cf710 0x7fa67c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.438+0000 7fa683d32700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa67c06a980 0x7fa67c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.438+0000 7fa683d32700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67c15cc10 con 0x7fa67c071890 2023-11-08T08:18:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.438+0000 7fa683d32700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c06c9b0 con 0x7fa67c06a980 2023-11-08T08:18:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa683d32700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c06cc90 con 0x7fa67c0cf710 2023-11-08T08:18:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa67c0cf710 0x7fa67c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa67c0cf710 0x7fa67c0ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57670/0 (socket says 172.21.15.129:57670) 2023-11-08T08:18:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 -- 172.21.15.129:0/563541230 learned_addr learned my addr 172.21.15.129:0/563541230 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:25.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa6822cf700 1 --2- 172.21.15.129:0/563541230 >> v2:172.21.15.116:3300/0 conn(0x7fa67c06a980 0x7fa67c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 -- 172.21.15.129:0/563541230 >> v2:172.21.15.116:3300/0 conn(0x7fa67c06a980 msgr2=0x7fa67c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:25.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 --2- 172.21.15.129:0/563541230 >> v2:172.21.15.116:3300/0 conn(0x7fa67c06a980 0x7fa67c071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 -- 172.21.15.129:0/563541230 >> v1:172.21.15.116:6789/0 conn(0x7fa67c071890 legacy=0x7fa67c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.444 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 -- 172.21.15.129:0/563541230 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c06cf70 con 0x7fa67c0cf710 2023-11-08T08:18:25.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa681ace700 1 --2- 172.21.15.129:0/563541230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c0cf710 0x7fa67c0ce740 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6780098f0 tx=0x7fa678009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca34a260e251f2f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:25.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa680acc700 1 -- 172.21.15.129:0/563541230 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678011830 con 0x7fa67c0cf710 2023-11-08T08:18:25.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa680acc700 1 -- 172.21.15.129:0/563541230 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa678011990 con 0x7fa67c0cf710 2023-11-08T08:18:25.445 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.439+0000 7fa680acc700 1 -- 172.21.15.129:0/563541230 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678011d20 con 0x7fa67c0cf710 2023-11-08T08:18:25.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 -- 172.21.15.129:0/563541230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c0cf710 msgr2=0x7fa67c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 --2- 172.21.15.129:0/563541230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c0cf710 0x7fa67c0ce740 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6780098f0 tx=0x7fa678009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:25.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 -- 172.21.15.129:0/563541230 shutdown_connections 2023-11-08T08:18:25.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 --2- 172.21.15.129:0/563541230 >> v2:172.21.15.116:3300/0 conn(0x7fa67c06a980 0x7fa67c071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.446 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 --2- 172.21.15.129:0/563541230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c0cf710 0x7fa67c0ce740 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 -- 172.21.15.129:0/563541230 >> 172.21.15.129:0/563541230 conn(0x7fa67c06c010 msgr2=0x7fa67c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:25.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 -- 172.21.15.129:0/563541230 shutdown_connections 2023-11-08T08:18:25.447 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.440+0000 7fa683d32700 1 -- 172.21.15.129:0/563541230 wait complete. 2023-11-08T08:18:25.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.441+0000 7fa683d32700 1 Processor -- start 2023-11-08T08:18:25.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.441+0000 7fa683d32700 1 -- start start 2023-11-08T08:18:25.451 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa683d32700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 0x7fa67c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa683d32700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa67c071890 0x7fa67c14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa683d32700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa67c0cf710 0x7fa67c0c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa683d32700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c0cac20 con 0x7fa67c071890 2023-11-08T08:18:25.452 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa683d32700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c15d760 con 0x7fa67c06a980 2023-11-08T08:18:25.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa683d32700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c15cea0 con 0x7fa67c0cf710 2023-11-08T08:18:25.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa681ace700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 0x7fa67c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa681ace700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 0x7fa67c0ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57684/0 (socket says 172.21.15.129:57684) 2023-11-08T08:18:25.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa681ace700 1 -- 172.21.15.129:0/805516984 learned_addr learned my addr 172.21.15.129:0/805516984 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:25.453 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.442+0000 7fa6822cf700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa67c0cf710 0x7fa67c0c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6812cd700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa67c071890 0x7fa67c14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa681ace700 1 -- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa67c0cf710 msgr2=0x7fa67c0c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa681ace700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa67c0cf710 0x7fa67c0c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.454 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa681ace700 1 -- 172.21.15.129:0/805516984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa67c071890 msgr2=0x7fa67c14fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa681ace700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa67c071890 0x7fa67c14fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa681ace700 1 -- 172.21.15.129:0/805516984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa678009550 con 0x7fa67c06a980 2023-11-08T08:18:25.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6822cf700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa67c0cf710 0x7fa67c0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:25.455 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa681ace700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 0x7fa67c0ca6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa678009130 tx=0x7fa678004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678004510 con 0x7fa67c06a980 2023-11-08T08:18:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67801e070 con 0x7fa67c06a980 2023-11-08T08:18:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6780038c0 con 0x7fa67c06a980 2023-11-08T08:18:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c06d530 con 0x7fa67c06a980 2023-11-08T08:18:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c06d250 con 0x7fa67c06a980 2023-11-08T08:18:25.457 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fa67800f050 con 0x7fa67c06a980 2023-11-08T08:18:25.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.443+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa67801a6f0 con 0x7fa67c06a980 2023-11-08T08:18:25.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.444+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c160480 con 0x7fa67c06a980 2023-11-08T08:18:25.458 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.446+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fa678032060 con 0x7fa67c06a980 2023-11-08T08:18:25.530 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.528+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "028d5009-e114-46ca-86e3-92451886a981", "id": 0} v 0) v1 -- 0x7fa67c0743f0 con 0x7fa67c06a980 2023-11-08T08:18:25.535 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.534+0000 7fa6727fc700 1 -- 172.21.15.129:0/805516984 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "028d5009-e114-46ca-86e3-92451886a981", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa678020070 con 0x7fa67c06a980 2023-11-08T08:18:25.536 INFO:teuthology.orchestra.run.smithi129.stdout:0 2023-11-08T08:18:25.537 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 msgr2=0x7fa67c0ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.537 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 0x7fa67c0ca6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa678009130 tx=0x7fa678004750 comp rx=0 tx=0).stop 2023-11-08T08:18:25.537 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 shutdown_connections 2023-11-08T08:18:25.537 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa67c0cf710 0x7fa67c0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.538 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa67c06a980 0x7fa67c0ca6e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.538 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 --2- 172.21.15.129:0/805516984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa67c071890 0x7fa67c14fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.538 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 >> 172.21.15.129:0/805516984 conn(0x7fa67c06c010 msgr2=0x7fa67c0b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:25.538 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 shutdown_connections 2023-11-08T08:18:25.539 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.535+0000 7fa683d32700 1 -- 172.21.15.129:0/805516984 wait complete. 2023-11-08T08:18:25.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 0a098844-7946-42be-b0b5-0799a4fe3407 1 2023-11-08T08:18:25.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.669+0000 7f8eec8b0700 1 Processor -- start 2023-11-08T08:18:25.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.669+0000 7f8eec8b0700 1 -- start start 2023-11-08T08:18:25.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8eec8b0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ee806a980 0x7f8ee80cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8eec8b0700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee815e0d0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8eec8b0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80ce560 con 0x7f8ee815a4d0 2023-11-08T08:18:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8eec8b0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee815e320 con 0x7f8ee806a980 2023-11-08T08:18:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee659c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8ee815a4d0 0x7f8ee806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57958/0 (socket says 172.21.15.129:57958) 2023-11-08T08:18:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee659c700 1 -- 172.21.15.129:0/4206402968 learned_addr learned my addr 172.21.15.129:0/4206402968 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-11-08T08:18:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee5d9b700 1 --2- 172.21.15.129:0/4206402968 >> v2:172.21.15.116:3300/0 conn(0x7f8ee806a980 0x7f8ee80cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634837457 0 0) 0x7f8ee80ce560 con 0x7f8ee815a4d0 2023-11-08T08:18:25.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed00035b0 con 0x7f8ee815a4d0 2023-11-08T08:18:25.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890656827 0 0) 0x7f8ee815e0d0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee80ce560 con 0x7f8ee80cdd50 2023-11-08T08:18:25.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/4206402968 >> v1:172.21.15.129:6789/0 conn(0x7f8ee815a4d0 legacy=0x7f8ee806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/4206402968 >> v1:172.21.15.116:6789/0 conn(0x7f8ee80cdd50 legacy=0x7f8ee80cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.670+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/4206402968 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee815e600 con 0x7f8ee806a980 2023-11-08T08:18:25.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.671+0000 7f8ee5d9b700 1 --2- 172.21.15.129:0/4206402968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80cd810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8edc009840 tx=0x7f8edc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ae35f689fc7a7c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:25.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.671+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc011830 con 0x7f8ee806a980 2023-11-08T08:18:25.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.671+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc011990 con 0x7f8ee806a980 2023-11-08T08:18:25.676 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.671+0000 7f8ee559a700 1 -- 172.21.15.129:0/4206402968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc011d20 con 0x7f8ee806a980 2023-11-08T08:18:25.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.671+0000 7f8eec8b0700 1 -- 172.21.15.129:0/4206402968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 msgr2=0x7f8ee80cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.671+0000 7f8eec8b0700 1 --2- 172.21.15.129:0/4206402968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80cd810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8edc009840 tx=0x7f8edc009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:25.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.672+0000 7f8eec8b0700 1 -- 172.21.15.129:0/4206402968 shutdown_connections 2023-11-08T08:18:25.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.672+0000 7f8eec8b0700 1 --2- 172.21.15.129:0/4206402968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80cd810 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.679 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.672+0000 7f8eec8b0700 1 -- 172.21.15.129:0/4206402968 >> 172.21.15.129:0/4206402968 conn(0x7f8ee806c010 msgr2=0x7f8ee80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.672+0000 7f8eec8b0700 1 -- 172.21.15.129:0/4206402968 shutdown_connections 2023-11-08T08:18:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.672+0000 7f8eec8b0700 1 -- 172.21.15.129:0/4206402968 wait complete. 2023-11-08T08:18:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.673+0000 7f8eec8b0700 1 Processor -- start 2023-11-08T08:18:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.673+0000 7f8eec8b0700 1 -- start start 2023-11-08T08:18:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8eec8b0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8eec8b0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 0x7f8ee815b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8eec8b0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ee815a4d0 0x7f8ee80c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8eec8b0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80c9bd0 con 0x7f8ee806a980 2023-11-08T08:18:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8eec8b0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80c98f0 con 0x7f8ee815a4d0 2023-11-08T08:18:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8eec8b0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee815ac80 con 0x7f8ee80cdd50 2023-11-08T08:18:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8ee5d9b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 0x7f8ee815b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8ee5d9b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 0x7f8ee815b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34412/0 (socket says 172.21.15.129:34412) 2023-11-08T08:18:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/3400978896 learned_addr learned my addr 172.21.15.129:0/3400978896 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee659c700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.674+0000 7f8ee6d9d700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ee815a4d0 0x7f8ee80c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ee815a4d0 msgr2=0x7f8ee80c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee5d9b700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ee815a4d0 0x7f8ee80c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/3400978896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 msgr2=0x7f8ee80c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee5d9b700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee5d9b700 1 -- 172.21.15.129:0/3400978896 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0094a0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ee5d9b700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 0x7f8ee815b760 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0090b0 tx=0x7f8edc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc01a6e0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc01a840 con 0x7f8ee80cdd50 2023-11-08T08:18:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee80ce1b0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc01abb0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee815e8e0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f8edc02b030 con 0x7f8ee80cdd50 2023-11-08T08:18:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.675+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f8edc02eb00 con 0x7f8ee80cdd50 2023-11-08T08:18:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.676+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee815bca0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.681+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f8edc029070 con 0x7f8ee80cdd50 2023-11-08T08:18:25.766 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.765+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0a098844-7946-42be-b0b5-0799a4fe3407", "id": 1} v 0) v1 -- 0x7f8ee815a8a0 con 0x7f8ee80cdd50 2023-11-08T08:18:25.774 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.773+0000 7f8ed6ffd700 1 -- 172.21.15.129:0/3400978896 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a098844-7946-42be-b0b5-0799a4fe3407", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8edc015020 con 0x7f8ee80cdd50 2023-11-08T08:18:25.774 INFO:teuthology.orchestra.run.smithi129.stdout:1 2023-11-08T08:18:25.776 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 msgr2=0x7f8ee815b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 0x7f8ee815b760 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0090b0 tx=0x7f8edc004750 comp rx=0 tx=0).stop 2023-11-08T08:18:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 shutdown_connections 2023-11-08T08:18:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ee815a4d0 0x7f8ee80c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee806a980 0x7f8ee80c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 --2- 172.21.15.129:0/3400978896 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ee80cdd50 0x7f8ee815b760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 >> 172.21.15.129:0/3400978896 conn(0x7f8ee806c010 msgr2=0x7f8ee80b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 shutdown_connections 2023-11-08T08:18:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.775+0000 7f8eec8b0700 1 -- 172.21.15.129:0/3400978896 wait complete. 2023-11-08T08:18:25.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new b092ec54-86b1-4d39-bbdc-6cb4d026ad59 2 2023-11-08T08:18:25.845 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:25.844+0000 7f8c088cb240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-08T08:18:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.914+0000 7f5d0ab54700 1 Processor -- start 2023-11-08T08:18:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.914+0000 7f5d0ab54700 1 -- start start 2023-11-08T08:18:25.917 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d0ab54700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5d040718e0 0x7f5d0406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.918 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d0ab54700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d04160a50 con 0x7f5d0406cdc0 2023-11-08T08:18:25.918 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d0ab54700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d04160d30 con 0x7f5d0415c810 2023-11-08T08:18:25.918 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d0ab54700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0415cbe0 con 0x7f5d040718e0 2023-11-08T08:18:25.918 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d03fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5d040718e0 0x7f5d0406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d088f0700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f5d0415c810 0x7f5d04160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57968/0 (socket says 172.21.15.129:57968) 2023-11-08T08:18:25.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d088f0700 1 -- 172.21.15.129:0/366817851 learned_addr learned my addr 172.21.15.129:0/366817851 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-11-08T08:18:25.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d03fff700 1 --2- 172.21.15.129:0/366817851 >> v2:172.21.15.129:3300/0 conn(0x7f5d040718e0 0x7f5d0406c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57704/0 (socket says 172.21.15.129:57704) 2023-11-08T08:18:25.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d03fff700 1 -- 172.21.15.129:0/366817851 >> v1:172.21.15.129:6789/0 conn(0x7f5d0415c810 legacy=0x7f5d04160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.919 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d03fff700 1 -- 172.21.15.129:0/366817851 >> v1:172.21.15.116:6789/0 conn(0x7f5d0406cdc0 legacy=0x7f5d0406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.920 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d03fff700 1 -- 172.21.15.129:0/366817851 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d04161010 con 0x7f5d040718e0 2023-11-08T08:18:25.920 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d03fff700 1 --2- 172.21.15.129:0/366817851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d0406c880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009870 tx=0x7f5cf0009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90cdf35ebf14568a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:25.920 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d037fe700 1 -- 172.21.15.129:0/366817851 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0011830 con 0x7f5d040718e0 2023-11-08T08:18:25.920 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d037fe700 1 -- 172.21.15.129:0/366817851 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf0011990 con 0x7f5d040718e0 2023-11-08T08:18:25.920 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.915+0000 7f5d037fe700 1 -- 172.21.15.129:0/366817851 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0011d20 con 0x7f5d040718e0 2023-11-08T08:18:25.921 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 -- 172.21.15.129:0/366817851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 msgr2=0x7f5d0406c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.921 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 --2- 172.21.15.129:0/366817851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d0406c880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009870 tx=0x7f5cf0009b80 comp rx=0 tx=0).stop 2023-11-08T08:18:25.921 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 -- 172.21.15.129:0/366817851 shutdown_connections 2023-11-08T08:18:25.921 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 --2- 172.21.15.129:0/366817851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d0406c880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 -- 172.21.15.129:0/366817851 >> 172.21.15.129:0/366817851 conn(0x7f5d0406c010 msgr2=0x7f5d040cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:25.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 -- 172.21.15.129:0/366817851 shutdown_connections 2023-11-08T08:18:25.922 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.916+0000 7f5d0ab54700 1 -- 172.21.15.129:0/366817851 wait complete. 2023-11-08T08:18:25.924 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.918+0000 7f5d0ab54700 1 Processor -- start 2023-11-08T08:18:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.918+0000 7f5d0ab54700 1 -- start start 2023-11-08T08:18:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0406cdc0 0x7f5d04153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d04153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d0415c810 0x7f5d041e6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d041e9130 con 0x7f5d0406cdc0 2023-11-08T08:18:25.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d041e9b80 con 0x7f5d040718e0 2023-11-08T08:18:25.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d040c3700 con 0x7f5d0415c810 2023-11-08T08:18:25.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d03fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d04153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d03fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d04153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57720/0 (socket says 172.21.15.129:57720) 2023-11-08T08:18:25.926 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d03fff700 1 -- 172.21.15.129:0/3583882854 learned_addr learned my addr 172.21.15.129:0/3583882854 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d0415c810 0x7f5d041e6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d088f0700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0406cdc0 0x7f5d04153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 -- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 msgr2=0x7f5d04153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d04153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 -- 172.21.15.129:0/3583882854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0406cdc0 msgr2=0x7f5d04153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0406cdc0 0x7f5d04153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 -- 172.21.15.129:0/3583882854 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf00094a0 con 0x7f5d0415c810 2023-11-08T08:18:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d03fff700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d04153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d090f1700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d0415c810 0x7f5d041e6420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d0406d1c0 tx=0x7f5cfc00ce70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc003c00 con 0x7f5d0415c810 2023-11-08T08:18:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc003d60 con 0x7f5d0415c810 2023-11-08T08:18:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d040ce740 con 0x7f5d0415c810 2023-11-08T08:18:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.919+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00a9c0 con 0x7f5d0415c810 2023-11-08T08:18:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.920+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d040ceb20 con 0x7f5d0415c810 2023-11-08T08:18:25.930 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.920+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f5cfc010050 con 0x7f5d0415c810 2023-11-08T08:18:25.930 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.920+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f5cfc005800 con 0x7f5d0415c810 2023-11-08T08:18:25.930 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.921+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d040ce8d0 con 0x7f5d0415c810 2023-11-08T08:18:25.930 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:25.921+0000 7f0604c66240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-08T08:18:25.931 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:25.923+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f5cfc0050d0 con 0x7f5d0415c810 2023-11-08T08:18:26.006 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.005+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b092ec54-86b1-4d39-bbdc-6cb4d026ad59", "id": 2} v 0) v1 -- 0x7f5d041e9320 con 0x7f5d0415c810 2023-11-08T08:18:26.014 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.013+0000 7f5d017fa700 1 -- 172.21.15.129:0/3583882854 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b092ec54-86b1-4d39-bbdc-6cb4d026ad59", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5cfc005350 con 0x7f5d0415c810 2023-11-08T08:18:26.014 INFO:teuthology.orchestra.run.smithi129.stdout:2 2023-11-08T08:18:26.015 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d0415c810 msgr2=0x7f5d041e6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d0415c810 0x7f5d041e6420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d0406d1c0 tx=0x7f5cfc00ce70 comp rx=0 tx=0).stop 2023-11-08T08:18:26.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 shutdown_connections 2023-11-08T08:18:26.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d0415c810 0x7f5d041e6420 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.016 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0406cdc0 0x7f5d04153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.017 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 --2- 172.21.15.129:0/3583882854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040718e0 0x7f5d04153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.017 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 >> 172.21.15.129:0/3583882854 conn(0x7f5d0406c010 msgr2=0x7f5d040ae0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.017 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 shutdown_connections 2023-11-08T08:18:26.017 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.014+0000 7f5d0ab54700 1 -- 172.21.15.129:0/3583882854 wait complete. 2023-11-08T08:18:26.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 081cf617-173d-4c8e-b5fa-60f71b817a59 3 2023-11-08T08:18:26.151 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.150+0000 7f69c011c700 1 Processor -- start 2023-11-08T08:18:26.151 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.150+0000 7f69c011c700 1 -- start start 2023-11-08T08:18:26.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69c011c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69b80d0ad0 0x7f69b80cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69c011c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b80cc740 con 0x7f69b80b2ee0 2023-11-08T08:18:26.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69c011c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b815a9c0 con 0x7f69b815a5f0 2023-11-08T08:18:26.154 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69c011c700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80c7b80 con 0x7f69b80d0ad0 2023-11-08T08:18:26.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69bdeb8700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f69b815a5f0 0x7f69b80b27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57980/0 (socket says 172.21.15.129:57980) 2023-11-08T08:18:26.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69bdeb8700 1 -- 172.21.15.129:0/2285858964 learned_addr learned my addr 172.21.15.129:0/2285858964 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-11-08T08:18:26.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69be6b9700 1 --2- 172.21.15.129:0/2285858964 >> v2:172.21.15.116:3300/0 conn(0x7f69b80d0ad0 0x7f69b80cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.155 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032981410 0 0) 0x7f69b815a9c0 con 0x7f69b815a5f0 2023-11-08T08:18:26.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a40035b0 con 0x7f69b815a5f0 2023-11-08T08:18:26.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783038123 0 0) 0x7f69b80cc740 con 0x7f69b80b2ee0 2023-11-08T08:18:26.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b815a9c0 con 0x7f69b80b2ee0 2023-11-08T08:18:26.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69be6b9700 1 -- 172.21.15.129:0/2285858964 >> v1:172.21.15.129:6789/0 conn(0x7f69b815a5f0 legacy=0x7f69b80b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.156 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69be6b9700 1 -- 172.21.15.129:0/2285858964 >> v1:172.21.15.116:6789/0 conn(0x7f69b80b2ee0 legacy=0x7f69b80ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69be6b9700 1 -- 172.21.15.129:0/2285858964 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80cc990 con 0x7f69b80d0ad0 2023-11-08T08:18:26.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.151+0000 7f69be6b9700 1 --2- 172.21.15.129:0/2285858964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80cc200 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3196cc92808082f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.157 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000e070 con 0x7f69b80d0ad0 2023-11-08T08:18:26.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000bb20 con 0x7f69b80d0ad0 2023-11-08T08:18:26.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69bceb6700 1 -- 172.21.15.129:0/2285858964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0012430 con 0x7f69b80d0ad0 2023-11-08T08:18:26.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 -- 172.21.15.129:0/2285858964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 msgr2=0x7f69b80cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.158 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 --2- 172.21.15.129:0/2285858964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80cc200 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b000b330 comp rx=0 tx=0).stop 2023-11-08T08:18:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 -- 172.21.15.129:0/2285858964 shutdown_connections 2023-11-08T08:18:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 --2- 172.21.15.129:0/2285858964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80cc200 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 -- 172.21.15.129:0/2285858964 >> 172.21.15.129:0/2285858964 conn(0x7f69b806ea80 msgr2=0x7f69b806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 -- 172.21.15.129:0/2285858964 shutdown_connections 2023-11-08T08:18:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.152+0000 7f69c011c700 1 -- 172.21.15.129:0/2285858964 wait complete. 2023-11-08T08:18:26.162 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.153+0000 7f69c011c700 1 Processor -- start 2023-11-08T08:18:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 -- start start 2023-11-08T08:18:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 0x7f69b814aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69b815a5f0 0x7f69b81534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b81e5bf0 con 0x7f69b80d0ad0 2023-11-08T08:18:26.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c94e0 con 0x7f69b80b2ee0 2023-11-08T08:18:26.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69c011c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c9d80 con 0x7f69b815a5f0 2023-11-08T08:18:26.164 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69bdeb8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 0x7f69b814aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69be6b9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69b815a5f0 0x7f69b81534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69bdeb8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 0x7f69b814aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57722/0 (socket says 172.21.15.129:57722) 2023-11-08T08:18:26.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69be6b9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69b815a5f0 0x7f69b81534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34432/0 (socket says 172.21.15.129:34432) 2023-11-08T08:18:26.165 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69bdeb8700 1 -- 172.21.15.129:0/1018796773 learned_addr learned my addr 172.21.15.129:0/1018796773 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.154+0000 7f69bd6b7700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bdeb8700 1 -- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69b815a5f0 msgr2=0x7f69b81534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bdeb8700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69b815a5f0 0x7f69b81534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bdeb8700 1 -- 172.21.15.129:0/1018796773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 msgr2=0x7f69b80c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bdeb8700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bdeb8700 1 -- 172.21.15.129:0/1018796773 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b0009a70 con 0x7f69b80b2ee0 2023-11-08T08:18:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bd6b7700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69bdeb8700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 0x7f69b814aa60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c730 tx=0x7f69b400ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4005d70 con 0x7f69b80b2ee0 2023-11-08T08:18:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b40070d0 con 0x7f69b80b2ee0 2023-11-08T08:18:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4007440 con 0x7f69b80b2ee0 2023-11-08T08:18:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b80ccf50 con 0x7f69b80b2ee0 2023-11-08T08:18:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b80ccc70 con 0x7f69b80b2ee0 2023-11-08T08:18:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f69b4019030 con 0x7f69b80b2ee0 2023-11-08T08:18:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.155+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f69b401c440 con 0x7f69b80b2ee0 2023-11-08T08:18:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.156+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b80cd230 con 0x7f69b80b2ee0 2023-11-08T08:18:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.160+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f69b4021060 con 0x7f69b80b2ee0 2023-11-08T08:18:26.187 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:26.186+0000 7f8c088cb240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-08T08:18:26.244 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.243+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "081cf617-173d-4c8e-b5fa-60f71b817a59", "id": 3} v 0) v1 -- 0x7f69b80743f0 con 0x7f69b80b2ee0 2023-11-08T08:18:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.250+0000 7f69ae7fc700 1 -- 172.21.15.129:0/1018796773 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "081cf617-173d-4c8e-b5fa-60f71b817a59", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f69b4017070 con 0x7f69b80b2ee0 2023-11-08T08:18:26.252 INFO:teuthology.orchestra.run.smithi129.stdout:3 2023-11-08T08:18:26.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 msgr2=0x7f69b814aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 0x7f69b814aa60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c730 tx=0x7f69b400ca40 comp rx=0 tx=0).stop 2023-11-08T08:18:26.253 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 shutdown_connections 2023-11-08T08:18:26.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f69b815a5f0 0x7f69b81534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69b80b2ee0 0x7f69b814aa60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 --2- 172.21.15.129:0/1018796773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80d0ad0 0x7f69b80c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.254 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 >> 172.21.15.129:0/1018796773 conn(0x7f69b806ea80 msgr2=0x7f69b8158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 shutdown_connections 2023-11-08T08:18:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.251+0000 7f69c011c700 1 -- 172.21.15.129:0/1018796773 wait complete. 2023-11-08T08:18:26.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new d55405eb-ebeb-432f-9309-8fe8e494663e 4 2023-11-08T08:18:26.302 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:26.301+0000 7f0604c66240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-08T08:18:26.385 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.384+0000 7ff67295f700 1 Processor -- start 2023-11-08T08:18:26.386 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.384+0000 7ff67295f700 1 -- start start 2023-11-08T08:18:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff67295f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 0x7ff66c160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff67295f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c06c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff67295f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c1607a0 con 0x7ff66c06cdc0 2023-11-08T08:18:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff67295f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff66c15cbe0 con 0x7ff66c15c810 2023-11-08T08:18:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff67295f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c160a50 con 0x7ff66c0718e0 2023-11-08T08:18:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 0x7ff66c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c06c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34436/0 (socket says 172.21.15.129:34436) 2023-11-08T08:18:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 0x7ff66c160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57734/0 (socket says 172.21.15.129:57734) 2023-11-08T08:18:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 -- 172.21.15.129:0/3849374816 learned_addr learned my addr 172.21.15.129:0/3849374816 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 -- 172.21.15.129:0/3849374816 >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 msgr2=0x7ff66c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 --2- 172.21.15.129:0/3849374816 >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 0x7ff66c160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 -- 172.21.15.129:0/3849374816 >> v1:172.21.15.116:6789/0 conn(0x7ff66c06cdc0 legacy=0x7ff66c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66b7fe700 1 -- 172.21.15.129:0/3849374816 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c160d30 con 0x7ff66c0718e0 2023-11-08T08:18:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.385+0000 7ff66bfff700 1 --2- 172.21.15.129:0/3849374816 >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 0x7ff66c160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff66b7fe700 1 --2- 172.21.15.129:0/3849374816 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c06c880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0098f0 tx=0x7ff65c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf163ad19c8f6f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff66affd700 1 -- 172.21.15.129:0/3849374816 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c011830 con 0x7ff66c0718e0 2023-11-08T08:18:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff66affd700 1 -- 172.21.15.129:0/3849374816 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c011990 con 0x7ff66c0718e0 2023-11-08T08:18:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff66affd700 1 -- 172.21.15.129:0/3849374816 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c011d20 con 0x7ff66c0718e0 2023-11-08T08:18:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 -- 172.21.15.129:0/3849374816 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 msgr2=0x7ff66c06c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 --2- 172.21.15.129:0/3849374816 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c06c880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0098f0 tx=0x7ff65c009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 -- 172.21.15.129:0/3849374816 shutdown_connections 2023-11-08T08:18:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 --2- 172.21.15.129:0/3849374816 >> v2:172.21.15.129:3300/0 conn(0x7ff66c15c810 0x7ff66c160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 --2- 172.21.15.129:0/3849374816 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c06c880 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 -- 172.21.15.129:0/3849374816 >> 172.21.15.129:0/3849374816 conn(0x7ff66c06c010 msgr2=0x7ff66c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.386+0000 7ff67295f700 1 -- 172.21.15.129:0/3849374816 shutdown_connections 2023-11-08T08:18:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.387+0000 7ff67295f700 1 -- 172.21.15.129:0/3849374816 wait complete. 2023-11-08T08:18:26.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.388+0000 7ff67295f700 1 Processor -- start 2023-11-08T08:18:26.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.388+0000 7ff67295f700 1 -- start start 2023-11-08T08:18:26.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff67295f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff66c06cdc0 0x7ff66c14f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.396 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff67295f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c0c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff67295f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c15c810 0x7ff66c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff67295f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0c6340 con 0x7ff66c15c810 2023-11-08T08:18:26.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff67295f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0c56c0 con 0x7ff66c06cdc0 2023-11-08T08:18:26.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff67295f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0c59a0 con 0x7ff66c0718e0 2023-11-08T08:18:26.397 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c0c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff66c06cdc0 0x7ff66c14f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c0c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34446/0 (socket says 172.21.15.129:34446) 2023-11-08T08:18:26.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66b7fe700 1 -- 172.21.15.129:0/3300840267 learned_addr learned my addr 172.21.15.129:0/3300840267 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:26.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff670efc700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c15c810 0x7ff66c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.398 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66bfff700 1 -- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 msgr2=0x7ff66c0c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66bfff700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c0c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66bfff700 1 -- 172.21.15.129:0/3300840267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c15c810 msgr2=0x7ff66c15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.389+0000 7ff66bfff700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c15c810 0x7ff66c15b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff66bfff700 1 -- 172.21.15.129:0/3300840267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540094a0 con 0x7ff66c06cdc0 2023-11-08T08:18:26.399 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff66b7fe700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:26.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff66bfff700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff66c06cdc0 0x7ff66c14f160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff65400e9b0 tx=0x7ff65400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65400ee50 con 0x7ff66c06cdc0 2023-11-08T08:18:26.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65400c850 con 0x7ff66c06cdc0 2023-11-08T08:18:26.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c009550 con 0x7ff66c06cdc0 2023-11-08T08:18:26.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65400cc00 con 0x7ff66c06cdc0 2023-11-08T08:18:26.400 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66c0ceb20 con 0x7ff66c06cdc0 2023-11-08T08:18:26.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7ff654019030 con 0x7ff66c06cdc0 2023-11-08T08:18:26.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.390+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7ff6540067b0 con 0x7ff66c06cdc0 2023-11-08T08:18:26.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.391+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66c161010 con 0x7ff66c06cdc0 2023-11-08T08:18:26.401 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.393+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7ff654021060 con 0x7ff66c06cdc0 2023-11-08T08:18:26.477 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.475+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d55405eb-ebeb-432f-9309-8fe8e494663e", "id": 4} v 0) v1 -- 0x7ff66c0ce740 con 0x7ff66c06cdc0 2023-11-08T08:18:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.483+0000 7ff668ff9700 1 -- 172.21.15.129:0/3300840267 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d55405eb-ebeb-432f-9309-8fe8e494663e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff654017070 con 0x7ff66c06cdc0 2023-11-08T08:18:26.484 INFO:teuthology.orchestra.run.smithi129.stdout:4 2023-11-08T08:18:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff66c06cdc0 msgr2=0x7ff66c14f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff66c06cdc0 0x7ff66c14f160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff65400e9b0 tx=0x7ff65400ecc0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 shutdown_connections 2023-11-08T08:18:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c15c810 0x7ff66c15b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff66c06cdc0 0x7ff66c14f160 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 --2- 172.21.15.129:0/3300840267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff66c0718e0 0x7ff66c0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 >> 172.21.15.129:0/3300840267 conn(0x7ff66c06c010 msgr2=0x7ff66c155d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 shutdown_connections 2023-11-08T08:18:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.484+0000 7ff67295f700 1 -- 172.21.15.129:0/3300840267 wait complete. 2023-11-08T08:18:26.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new a8ddead2-5aa5-4b7f-b6a0-01a42c452b48 5 2023-11-08T08:18:26.659 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.658+0000 7f3b18dba700 1 Processor -- start 2023-11-08T08:18:26.659 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.658+0000 7f3b18dba700 1 -- start start 2023-11-08T08:18:26.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b18dba700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 0x7f3b140b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b18dba700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b141599b0 0x7f3b1415d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.662 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b18dba700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b14159ec0 con 0x7f3b140af0a0 2023-11-08T08:18:26.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b18dba700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1415aa60 con 0x7f3b141599b0 2023-11-08T08:18:26.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b18dba700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140b1f60 con 0x7f3b140b1650 2023-11-08T08:18:26.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 0x7f3b140b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 0x7f3b140b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34462/0 (socket says 172.21.15.129:34462) 2023-11-08T08:18:26.663 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b12d9d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3b140af0a0 0x7f3b1415a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57986/0 (socket says 172.21.15.129:57986) 2023-11-08T08:18:26.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 -- 172.21.15.129:0/3699716235 learned_addr learned my addr 172.21.15.129:0/3699716235 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:26.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b11d9b700 1 --2- 172.21.15.129:0/3699716235 >> v2:172.21.15.116:3300/0 conn(0x7f3b141599b0 0x7f3b1415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1159a700 1 -- 172.21.15.129:0/3699716235 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284747651 0 0) 0x7f3b14159ec0 con 0x7f3b140af0a0 2023-11-08T08:18:26.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1159a700 1 -- 172.21.15.129:0/3699716235 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3afc0035b0 con 0x7f3b140af0a0 2023-11-08T08:18:26.664 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 -- 172.21.15.129:0/3699716235 >> v2:172.21.15.116:3300/0 conn(0x7f3b141599b0 msgr2=0x7f3b1415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:26.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 --2- 172.21.15.129:0/3699716235 >> v2:172.21.15.116:3300/0 conn(0x7f3b141599b0 0x7f3b1415d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 -- 172.21.15.129:0/3699716235 >> v1:172.21.15.129:6789/0 conn(0x7f3b140af0a0 legacy=0x7f3b1415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 -- 172.21.15.129:0/3699716235 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1415d8a0 con 0x7f3b140b1650 2023-11-08T08:18:26.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.659+0000 7f3b1259c700 1 --2- 172.21.15.129:0/3699716235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 0x7f3b140b1a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3b040098f0 tx=0x7f3b04009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ee2a72a090f8388 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.665 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b1159a700 1 -- 172.21.15.129:0/3699716235 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04011830 con 0x7f3b140b1650 2023-11-08T08:18:26.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b1159a700 1 -- 172.21.15.129:0/3699716235 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b04011990 con 0x7f3b140b1650 2023-11-08T08:18:26.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b1159a700 1 -- 172.21.15.129:0/3699716235 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04011d20 con 0x7f3b140b1650 2023-11-08T08:18:26.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 -- 172.21.15.129:0/3699716235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 msgr2=0x7f3b140b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 --2- 172.21.15.129:0/3699716235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 0x7f3b140b1a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3b040098f0 tx=0x7f3b04009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:26.666 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 -- 172.21.15.129:0/3699716235 shutdown_connections 2023-11-08T08:18:26.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 --2- 172.21.15.129:0/3699716235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140b1650 0x7f3b140b1a20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 --2- 172.21.15.129:0/3699716235 >> v2:172.21.15.116:3300/0 conn(0x7f3b141599b0 0x7f3b1415d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 -- 172.21.15.129:0/3699716235 >> 172.21.15.129:0/3699716235 conn(0x7f3b1406c010 msgr2=0x7f3b140cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 -- 172.21.15.129:0/3699716235 shutdown_connections 2023-11-08T08:18:26.667 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.660+0000 7f3b18dba700 1 -- 172.21.15.129:0/3699716235 wait complete. 2023-11-08T08:18:26.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 Processor -- start 2023-11-08T08:18:26.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 -- start start 2023-11-08T08:18:26.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 0x7f3b14148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.670 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b140b1650 0x7f3b14148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b140cbd40 0x7f3b14155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140b3410 con 0x7f3b140b1650 2023-11-08T08:18:26.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14149740 con 0x7f3b140cbd40 2023-11-08T08:18:26.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b18dba700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14149a20 con 0x7f3b140af0a0 2023-11-08T08:18:26.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b1259c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 0x7f3b14148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.671 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b12d9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b140cbd40 0x7f3b14155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b1259c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 0x7f3b14148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34466/0 (socket says 172.21.15.129:34466) 2023-11-08T08:18:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b1259c700 1 -- 172.21.15.129:0/951958973 learned_addr learned my addr 172.21.15.129:0/951958973 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.662+0000 7f3b11d9b700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b140b1650 0x7f3b14148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b1259c700 1 -- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b140cbd40 msgr2=0x7f3b14155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b1259c700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b140cbd40 0x7f3b14155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b1259c700 1 -- 172.21.15.129:0/951958973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b140b1650 msgr2=0x7f3b14148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b1259c700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b140b1650 0x7f3b14148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b1259c700 1 -- 172.21.15.129:0/951958973 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04009550 con 0x7f3b140af0a0 2023-11-08T08:18:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b1259c700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 0x7f3b14148980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3b04009c40 tx=0x7f3b04004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04004510 con 0x7f3b140af0a0 2023-11-08T08:18:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0401e070 con 0x7f3b140af0a0 2023-11-08T08:18:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1415de60 con 0x7f3b140af0a0 2023-11-08T08:18:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040038e0 con 0x7f3b140af0a0 2023-11-08T08:18:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1415db80 con 0x7f3b140af0a0 2023-11-08T08:18:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f3b0400f050 con 0x7f3b140af0a0 2023-11-08T08:18:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.663+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f3b04020070 con 0x7f3b140af0a0 2023-11-08T08:18:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.664+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1415d590 con 0x7f3b140af0a0 2023-11-08T08:18:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.666+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f3b04044b00 con 0x7f3b140af0a0 2023-11-08T08:18:26.749 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.748+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a8ddead2-5aa5-4b7f-b6a0-01a42c452b48", "id": 5} v 0) v1 -- 0x7f3b140743f0 con 0x7f3b140af0a0 2023-11-08T08:18:26.757 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.756+0000 7f3b02ffd700 1 -- 172.21.15.129:0/951958973 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8ddead2-5aa5-4b7f-b6a0-01a42c452b48", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3b04033070 con 0x7f3b140af0a0 2023-11-08T08:18:26.758 INFO:teuthology.orchestra.run.smithi129.stdout:5 2023-11-08T08:18:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 msgr2=0x7f3b14148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 0x7f3b14148980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3b04009c40 tx=0x7f3b04004750 comp rx=0 tx=0).stop 2023-11-08T08:18:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 shutdown_connections 2023-11-08T08:18:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b140cbd40 0x7f3b14155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3b140af0a0 0x7f3b14148980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 --2- 172.21.15.129:0/951958973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b140b1650 0x7f3b14148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 >> 172.21.15.129:0/951958973 conn(0x7f3b1406c010 msgr2=0x7f3b14071550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 shutdown_connections 2023-11-08T08:18:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.757+0000 7f3b18dba700 1 -- 172.21.15.129:0/951958973 wait complete. 2023-11-08T08:18:26.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new e053dced-1e55-4ff7-ab49-72dd3f415e5e 6 2023-11-08T08:18:26.891 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.890+0000 7fc585c83700 1 Processor -- start 2023-11-08T08:18:26.892 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.890+0000 7fc585c83700 1 -- start start 2023-11-08T08:18:26.894 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc585c83700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc580071890 0x7fc5800ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.894 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc585c83700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58015cc10 con 0x7fc58006a980 2023-11-08T08:18:26.894 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc585c83700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58006c9b0 con 0x7fc5800cf710 2023-11-08T08:18:26.895 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc585c83700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58006cc90 con 0x7fc580071890 2023-11-08T08:18:26.895 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57ffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc580071890 0x7fc5800ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.895 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57f7fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc5800cf710 0x7fc58006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57988/0 (socket says 172.21.15.129:57988) 2023-11-08T08:18:26.895 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57f7fe700 1 -- 172.21.15.129:0/2058007399 learned_addr learned my addr 172.21.15.129:0/2058007399 (peer_addr_for_me v1:172.21.15.129:0/0) 2023-11-08T08:18:26.895 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57ffff700 1 -- 172.21.15.129:0/2058007399 >> v1:172.21.15.129:6789/0 conn(0x7fc5800cf710 legacy=0x7fc58006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.896 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57ffff700 1 -- 172.21.15.129:0/2058007399 >> v1:172.21.15.116:6789/0 conn(0x7fc58006a980 legacy=0x7fc580160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.896 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57ffff700 1 -- 172.21.15.129:0/2058007399 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58006cf70 con 0x7fc580071890 2023-11-08T08:18:26.896 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57ffff700 1 --2- 172.21.15.129:0/2058007399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc580071890 0x7fc5800ce740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc578009d90 tx=0x7fc57800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a63fc318af4fa00 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.896 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57e7fc700 1 -- 172.21.15.129:0/2058007399 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57800e070 con 0x7fc580071890 2023-11-08T08:18:26.896 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.891+0000 7fc57e7fc700 1 -- 172.21.15.129:0/2058007399 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57800bbd0 con 0x7fc580071890 2023-11-08T08:18:26.897 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc57e7fc700 1 -- 172.21.15.129:0/2058007399 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578012430 con 0x7fc580071890 2023-11-08T08:18:26.897 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 -- 172.21.15.129:0/2058007399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc580071890 msgr2=0x7fc5800ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.897 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 --2- 172.21.15.129:0/2058007399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc580071890 0x7fc5800ce740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc578009d90 tx=0x7fc57800b330 comp rx=0 tx=0).stop 2023-11-08T08:18:26.897 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 -- 172.21.15.129:0/2058007399 shutdown_connections 2023-11-08T08:18:26.897 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 --2- 172.21.15.129:0/2058007399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc580071890 0x7fc5800ce740 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 -- 172.21.15.129:0/2058007399 >> 172.21.15.129:0/2058007399 conn(0x7fc58006c010 msgr2=0x7fc5800cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 -- 172.21.15.129:0/2058007399 shutdown_connections 2023-11-08T08:18:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.892+0000 7fc585c83700 1 -- 172.21.15.129:0/2058007399 wait complete. 2023-11-08T08:18:26.901 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.894+0000 7fc585c83700 1 Processor -- start 2023-11-08T08:18:26.901 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.894+0000 7fc585c83700 1 -- start start 2023-11-08T08:18:26.901 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc585c83700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 0x7fc580149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.901 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc585c83700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580071890 0x7fc5800b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.902 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc585c83700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5800cf710 0x7fc5800c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:26.902 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc585c83700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800c4f80 con 0x7fc580071890 2023-11-08T08:18:26.902 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc585c83700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800c4370 con 0x7fc5800cf710 2023-11-08T08:18:26.902 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc585c83700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800c4650 con 0x7fc58006a980 2023-11-08T08:18:26.902 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.895+0000 7fc57f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 0x7fc580149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.903 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5800cf710 0x7fc5800c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.903 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 0x7fc580149190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34472/0 (socket says 172.21.15.129:34472) 2023-11-08T08:18:26.903 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57f7fe700 1 -- 172.21.15.129:0/3150524885 learned_addr learned my addr 172.21.15.129:0/3150524885 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:26.903 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57effd700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580071890 0x7fc5800b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:26.903 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 -- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 msgr2=0x7fc580149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.904 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 0x7fc580149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.904 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 -- 172.21.15.129:0/3150524885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580071890 msgr2=0x7fc5800b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.904 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580071890 0x7fc5800b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.904 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 -- 172.21.15.129:0/3150524885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc578009a70 con 0x7fc5800cf710 2023-11-08T08:18:26.904 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57f7fe700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 0x7fc580149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:26.905 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc57ffff700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5800cf710 0x7fc5800c5410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc578016000 tx=0x7fc578004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:26.905 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578012da0 con 0x7fc5800cf710 2023-11-08T08:18:26.905 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578020070 con 0x7fc5800cf710 2023-11-08T08:18:26.905 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58006d530 con 0x7fc5800cf710 2023-11-08T08:18:26.905 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578004280 con 0x7fc5800cf710 2023-11-08T08:18:26.905 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58006d250 con 0x7fc5800cf710 2023-11-08T08:18:26.906 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fc57802e070 con 0x7fc5800cf710 2023-11-08T08:18:26.906 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.896+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fc578022070 con 0x7fc5800cf710 2023-11-08T08:18:26.906 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.897+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5800c4790 con 0x7fc5800cf710 2023-11-08T08:18:26.906 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.899+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fc5800c4790 con 0x7fc5800cf710 2023-11-08T08:18:26.907 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:26.899+0000 7f8c088cb240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-08T08:18:26.972 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:26.971+0000 7f8c088cb240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-08T08:18:26.984 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.982+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e053dced-1e55-4ff7-ab49-72dd3f415e5e", "id": 6} v 0) v1 -- 0x7fc5800743f0 con 0x7fc5800cf710 2023-11-08T08:18:26.989 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.988+0000 7fc567fff700 1 -- 172.21.15.129:0/3150524885 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e053dced-1e55-4ff7-ab49-72dd3f415e5e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc57802c060 con 0x7fc5800cf710 2023-11-08T08:18:26.990 INFO:teuthology.orchestra.run.smithi129.stdout:6 2023-11-08T08:18:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5800cf710 msgr2=0x7fc5800c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5800cf710 0x7fc5800c5410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc578016000 tx=0x7fc578004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 shutdown_connections 2023-11-08T08:18:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5800cf710 0x7fc5800c5410 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc58006a980 0x7fc580149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 --2- 172.21.15.129:0/3150524885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580071890 0x7fc5800b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 >> 172.21.15.129:0/3150524885 conn(0x7fc58006c010 msgr2=0x7fc58015c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 shutdown_connections 2023-11-08T08:18:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:26.989+0000 7fc585c83700 1 -- 172.21.15.129:0/3150524885 wait complete. 2023-11-08T08:18:27.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 87241c04-a052-4342-80f0-139e223046f6 7 2023-11-08T08:18:27.073 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.072+0000 7f0604c66240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-08T08:18:27.104 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.103+0000 7f8c088cb240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-08T08:18:27.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.124+0000 7f9e1b5cd700 1 Processor -- start 2023-11-08T08:18:27.125 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.124+0000 7f9e1b5cd700 1 -- start start 2023-11-08T08:18:27.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e1b5cd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e1415a5f0 0x7f9e140d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e1b5cd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9e140cc200 0x7f9e140b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e1b5cd700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1415a9c0 con 0x7f9e140b2d10 2023-11-08T08:18:27.128 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e1b5cd700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140b3190 con 0x7f9e1415a5f0 2023-11-08T08:18:27.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e1b5cd700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140c7b80 con 0x7f9e140cc200 2023-11-08T08:18:27.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9e140cc200 0x7f9e140b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9e140cc200 0x7f9e140b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57778/0 (socket says 172.21.15.129:57778) 2023-11-08T08:18:27.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 -- 172.21.15.129:0/2336802891 learned_addr learned my addr 172.21.15.129:0/2336802891 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:27.129 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e19369700 1 --2- 172.21.15.129:0/2336802891 >> v2:172.21.15.116:3300/0 conn(0x7f9e1415a5f0 0x7f9e140d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 -- 172.21.15.129:0/2336802891 >> v2:172.21.15.116:3300/0 conn(0x7f9e1415a5f0 msgr2=0x7f9e140d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:27.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 --2- 172.21.15.129:0/2336802891 >> v2:172.21.15.116:3300/0 conn(0x7f9e1415a5f0 0x7f9e140d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 -- 172.21.15.129:0/2336802891 >> v1:172.21.15.116:6789/0 conn(0x7f9e140b2d10 legacy=0x7f9e1415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 -- 172.21.15.129:0/2336802891 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e140cc990 con 0x7f9e140cc200 2023-11-08T08:18:27.130 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.125+0000 7f9e18b68700 1 --2- 172.21.15.129:0/2336802891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e140cc200 0x7f9e140b27d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9e1406d8d0 tx=0x7f9e04009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31166b58a99456c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:27.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e0bfff700 1 -- 172.21.15.129:0/2336802891 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04011830 con 0x7f9e140cc200 2023-11-08T08:18:27.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e0bfff700 1 -- 172.21.15.129:0/2336802891 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04011990 con 0x7f9e140cc200 2023-11-08T08:18:27.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e0bfff700 1 -- 172.21.15.129:0/2336802891 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04011d20 con 0x7f9e140cc200 2023-11-08T08:18:27.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/2336802891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e140cc200 msgr2=0x7f9e140b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.131 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/2336802891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e140cc200 0x7f9e140b27d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9e1406d8d0 tx=0x7f9e04009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/2336802891 shutdown_connections 2023-11-08T08:18:27.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/2336802891 >> v2:172.21.15.116:3300/0 conn(0x7f9e1415a5f0 0x7f9e140d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/2336802891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e140cc200 0x7f9e140b27d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/2336802891 >> 172.21.15.129:0/2336802891 conn(0x7f9e1406ea80 msgr2=0x7f9e1406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:27.132 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/2336802891 shutdown_connections 2023-11-08T08:18:27.133 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.126+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/2336802891 wait complete. 2023-11-08T08:18:27.135 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.127+0000 7f9e1b5cd700 1 Processor -- start 2023-11-08T08:18:27.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 -- start start 2023-11-08T08:18:27.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140b2d10 0x7f9e14154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 0x7f9e141e4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e1415a5f0 0x7f9e141e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.136 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141547b0 con 0x7f9e140b2d10 2023-11-08T08:18:27.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141e5d70 con 0x7f9e1415a5f0 2023-11-08T08:18:27.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e1b5cd700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140c7fc0 con 0x7f9e140cc200 2023-11-08T08:18:27.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e18b68700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 0x7f9e141e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.137 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e18b68700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 0x7f9e141e4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34478/0 (socket says 172.21.15.129:34478) 2023-11-08T08:18:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e18b68700 1 -- 172.21.15.129:0/3826527242 learned_addr learned my addr 172.21.15.129:0/3826527242 (peer_addr_for_me v2:172.21.15.129:0/0) 2023-11-08T08:18:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e19369700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140b2d10 0x7f9e14154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.128+0000 7f9e19b6a700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e1415a5f0 0x7f9e141e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e18b68700 1 -- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e1415a5f0 msgr2=0x7f9e141e4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e18b68700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e1415a5f0 0x7f9e141e4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e18b68700 1 -- 172.21.15.129:0/3826527242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140b2d10 msgr2=0x7f9e14154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e18b68700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140b2d10 0x7f9e14154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e18b68700 1 -- 172.21.15.129:0/3826527242 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04009550 con 0x7f9e140cc200 2023-11-08T08:18:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e19369700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140b2d10 0x7f9e14154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e18b68700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 0x7f9e141e4960 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e04004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04004380 con 0x7f9e140cc200 2023-11-08T08:18:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0401f070 con 0x7f9e140cc200 2023-11-08T08:18:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e140ccf50 con 0x7f9e140cc200 2023-11-08T08:18:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e040036a0 con 0x7f9e140cc200 2023-11-08T08:18:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e140ccc70 con 0x7f9e140cc200 2023-11-08T08:18:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f9e0402a020 con 0x7f9e140cc200 2023-11-08T08:18:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.129+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9e0401ac40 con 0x7f9e140cc200 2023-11-08T08:18:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.130+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e141e5580 con 0x7f9e140cc200 2023-11-08T08:18:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.132+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9e04015020 con 0x7f9e140cc200 2023-11-08T08:18:27.149 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.148+0000 7f0604c66240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-08T08:18:27.173 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.172+0000 7f8c088cb240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-08T08:18:27.216 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.215+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd new", "uuid": "87241c04-a052-4342-80f0-139e223046f6", "id": 7} v 0) v1 -- 0x7f9e140cd230 con 0x7f9e140cc200 2023-11-08T08:18:27.219 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.218+0000 7f9e09ffb700 1 -- 172.21.15.129:0/3826527242 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87241c04-a052-4342-80f0-139e223046f6", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9e0401a710 con 0x7f9e140cc200 2023-11-08T08:18:27.220 INFO:teuthology.orchestra.run.smithi129.stdout:7 2023-11-08T08:18:27.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 msgr2=0x7f9e141e4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 0x7f9e141e4960 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e04004750 comp rx=0 tx=0).stop 2023-11-08T08:18:27.221 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 shutdown_connections 2023-11-08T08:18:27.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e1415a5f0 0x7f9e141e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140b2d10 0x7f9e14154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 --2- 172.21.15.129:0/3826527242 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9e140cc200 0x7f9e141e4960 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.222 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 >> 172.21.15.129:0/3826527242 conn(0x7f9e1406ea80 msgr2=0x7f9e140b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:27.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 shutdown_connections 2023-11-08T08:18:27.223 INFO:teuthology.orchestra.run.smithi129.stderr:2023-11-08T08:18:27.219+0000 7f9e1b5cd700 1 -- 172.21.15.129:0/3826527242 wait complete. 2023-11-08T08:18:27.233 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-08T08:18:27.233 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-08T08:18:27.235 INFO:tasks.ceph.osd.0:Started 2023-11-08T08:18:27.236 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-08T08:18:27.236 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-08T08:18:27.238 INFO:tasks.ceph.osd.1:Started 2023-11-08T08:18:27.239 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-08T08:18:27.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-08T08:18:27.241 INFO:tasks.ceph.osd.2:Started 2023-11-08T08:18:27.242 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-08T08:18:27.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-08T08:18:27.244 INFO:tasks.ceph.osd.3:Started 2023-11-08T08:18:27.244 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-08T08:18:27.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-08T08:18:27.247 INFO:tasks.ceph.osd.4:Started 2023-11-08T08:18:27.247 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-08T08:18:27.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-08T08:18:27.249 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.247+0000 7f8c088cb240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-08T08:18:27.250 INFO:tasks.ceph.osd.5:Started 2023-11-08T08:18:27.251 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-08T08:18:27.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-08T08:18:27.253 INFO:tasks.ceph.osd.6:Started 2023-11-08T08:18:27.254 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-08T08:18:27.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-08T08:18:27.256 INFO:tasks.ceph.osd.7:Started 2023-11-08T08:18:27.257 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:27.305 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.304+0000 7f0604c66240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-08T08:18:27.394 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.393+0000 7f0604c66240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-08T08:18:27.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.413+0000 7fa418576700 1 Processor -- start 2023-11-08T08:18:27.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.413+0000 7fa418576700 1 -- start start 2023-11-08T08:18:27.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa418576700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa410071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa418576700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 0x7fa41006b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa418576700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4100cfb00 con 0x7fa4100b4a30 2023-11-08T08:18:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa418576700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41015aa20 con 0x7fa4100cff90 2023-11-08T08:18:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa418576700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100cf4d0 con 0x7fa4100b4660 2023-11-08T08:18:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416312700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa410071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa415b11700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 0x7fa41006b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416312700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa410071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:45702/0 (socket says 172.21.15.116:45702) 2023-11-08T08:18:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa415b11700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 0x7fa41006b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:52472/0 (socket says 172.21.15.116:52472) 2023-11-08T08:18:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa415b11700 1 -- 172.21.15.116:0/3596889764 learned_addr learned my addr 172.21.15.116:0/3596889764 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416b13700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa4100b4a30 0x7fa4100ac1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:36304/0 (socket says 172.21.15.116:36304) 2023-11-08T08:18:27.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416312700 1 -- 172.21.15.116:0/3596889764 >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 msgr2=0x7fa41006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:27.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416312700 1 --2- 172.21.15.116:0/3596889764 >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 0x7fa41006b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416312700 1 -- 172.21.15.116:0/3596889764 >> v1:172.21.15.129:6789/0 conn(0x7fa4100b4a30 legacy=0x7fa4100ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa416312700 1 -- 172.21.15.116:0/3596889764 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410072550 con 0x7fa4100b4660 2023-11-08T08:18:27.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa415b11700 1 --2- 172.21.15.116:0/3596889764 >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 0x7fa41006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.415+0000 7fa415310700 1 -- 172.21.15.116:0/3596889764 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233926559 0 0) 0x7fa4100cfb00 con 0x7fa4100b4a30 2023-11-08T08:18:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa416312700 1 --2- 172.21.15.116:0/3596889764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa410071f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa408009840 tx=0x7fa408009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ae5af37cf3cd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa415310700 1 -- 172.21.15.116:0/3596889764 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408011830 con 0x7fa4100b4660 2023-11-08T08:18:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa415310700 1 -- 172.21.15.116:0/3596889764 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa408011990 con 0x7fa4100b4660 2023-11-08T08:18:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa415310700 1 -- 172.21.15.116:0/3596889764 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408011d20 con 0x7fa4100b4660 2023-11-08T08:18:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 -- 172.21.15.116:0/3596889764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 msgr2=0x7fa410071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 --2- 172.21.15.116:0/3596889764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa410071f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa408009840 tx=0x7fa408009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:27.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 -- 172.21.15.116:0/3596889764 shutdown_connections 2023-11-08T08:18:27.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 --2- 172.21.15.116:0/3596889764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa410071f70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 --2- 172.21.15.116:0/3596889764 >> v2:172.21.15.129:3300/0 conn(0x7fa4100cff90 0x7fa41006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 -- 172.21.15.116:0/3596889764 >> 172.21.15.116:0/3596889764 conn(0x7fa4100b2fc0 msgr2=0x7fa410071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:27.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 -- 172.21.15.116:0/3596889764 shutdown_connections 2023-11-08T08:18:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.416+0000 7fa418576700 1 -- 172.21.15.116:0/3596889764 wait complete. 2023-11-08T08:18:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 Processor -- start 2023-11-08T08:18:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 -- start start 2023-11-08T08:18:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa4101f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4100b4a30 0x7fa4100cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4100cff90 0x7fa4100cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41015ab60 con 0x7fa4100cff90 2023-11-08T08:18:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101f1e10 con 0x7fa4100b4a30 2023-11-08T08:18:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.418+0000 7fa418576700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100ce5a0 con 0x7fa4100b4660 2023-11-08T08:18:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa4101f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416b13700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4100cff90 0x7fa4100cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa4101f18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:45716/0 (socket says 172.21.15.116:45716) 2023-11-08T08:18:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 -- 172.21.15.116:0/1584842446 learned_addr learned my addr 172.21.15.116:0/1584842446 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa415b11700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4100b4a30 0x7fa4100cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 -- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4100b4a30 msgr2=0x7fa4100cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4100b4a30 0x7fa4100cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 -- 172.21.15.116:0/1584842446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4100cff90 msgr2=0x7fa4100cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4100cff90 0x7fa4100cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 -- 172.21.15.116:0/1584842446 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400009550 con 0x7fa4100b4660 2023-11-08T08:18:27.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416b13700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4100cff90 0x7fa4100cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:27.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa416312700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa4101f18d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa4080090b0 tx=0x7fa408004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4080036a0 con 0x7fa4100b4660 2023-11-08T08:18:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40801f070 con 0x7fa4100b4660 2023-11-08T08:18:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408003a20 con 0x7fa4100b4660 2023-11-08T08:18:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4080094a0 con 0x7fa4100b4660 2023-11-08T08:18:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41006bad0 con 0x7fa4100b4660 2023-11-08T08:18:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fa40800f050 con 0x7fa4100b4660 2023-11-08T08:18:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.419+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa408021070 con 0x7fa4100b4660 2023-11-08T08:18:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.420+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4100cf790 con 0x7fa4100b4660 2023-11-08T08:18:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.422+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fa40802b020 con 0x7fa4100b4660 2023-11-08T08:18:27.456 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.455+0000 7f8c088cb240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-08T08:18:27.468 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.467+0000 7f0604c66240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-08T08:18:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.490+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4100b4eb0 con 0x7fa4100b4660 2023-11-08T08:18:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.491+0000 7fa406ffd700 1 -- 172.21.15.116:0/1584842446 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa408015020 con 0x7fa4100b4660 2023-11-08T08:18:27.493 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:27.493 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:27.217106+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 msgr2=0x7fa4101f18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa4101f18d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa4080090b0 tx=0x7fa408004750 comp rx=0 tx=0).stop 2023-11-08T08:18:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 shutdown_connections 2023-11-08T08:18:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4100cff90 0x7fa4100cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4100b4660 0x7fa4101f18d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 --2- 172.21.15.116:0/1584842446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4100b4a30 0x7fa4100cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 >> 172.21.15.116:0/1584842446 conn(0x7fa4100b2fc0 msgr2=0x7fa41006d580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 shutdown_connections 2023-11-08T08:18:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:27.492+0000 7fa418576700 1 -- 172.21.15.116:0/1584842446 wait complete. 2023-11-08T08:18:27.505 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-08T08:18:27.506 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-08T08:18:27.535 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.534+0000 7f8c088cb240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-08T08:18:27.614 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.613+0000 7f8c088cb240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-08T08:18:27.654 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.652+0000 7f0604c66240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-08T08:18:27.731 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.730+0000 7f0604c66240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-08T08:18:27.792 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.791+0000 7f8c088cb240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-08T08:18:27.812 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.811+0000 7f0604c66240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-08T08:18:27.849 INFO:tasks.ceph.osd.4.smithi129.stderr:2023-11-08T08:18:27.848+0000 7f5b935cf680 -1 Falling back to public interface 2023-11-08T08:18:27.861 INFO:tasks.ceph.osd.5.smithi129.stderr:2023-11-08T08:18:27.860+0000 7fa1fc655680 -1 Falling back to public interface 2023-11-08T08:18:27.862 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-08T08:18:27.861+0000 7f5158a94680 -1 Falling back to public interface 2023-11-08T08:18:27.867 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-08T08:18:27.866+0000 7fa114f9f680 -1 Falling back to public interface 2023-11-08T08:18:27.868 INFO:tasks.ceph.osd.7.smithi129.stderr:2023-11-08T08:18:27.867+0000 7fec40365680 -1 Falling back to public interface 2023-11-08T08:18:27.870 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-08T08:18:27.869+0000 7f0270960680 -1 Falling back to public interface 2023-11-08T08:18:27.876 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:27.875+0000 7f8c088cb240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-08T08:18:27.876 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-08T08:18:27.875+0000 7f14be7d5680 -1 Falling back to public interface 2023-11-08T08:18:27.881 INFO:tasks.ceph.osd.6.smithi129.stderr:2023-11-08T08:18:27.880+0000 7f4313eb1680 -1 Falling back to public interface 2023-11-08T08:18:27.993 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:27.991+0000 7f0604c66240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-08T08:18:28.073 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:28.072+0000 7f0604c66240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-08T08:18:28.159 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:28.158+0000 7f8c088cb240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-08T08:18:28.229 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:28.228+0000 7f8c088cb240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-08T08:18:28.313 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:28.364 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:28.363+0000 7f0604c66240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-08T08:18:28.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.425+0000 7fe194047700 1 Processor -- start 2023-11-08T08:18:28.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.425+0000 7fe194047700 1 -- start start 2023-11-08T08:18:28.428 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe194047700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe18c15a5f0 0x7fe18c0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:28.428 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe194047700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe18c0b2ee0 0x7fe18c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:28.428 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe194047700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18c15a9c0 con 0x7fe18c0cc200 2023-11-08T08:18:28.429 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe194047700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c0c7b80 con 0x7fe18c0b2ee0 2023-11-08T08:18:28.429 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe194047700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c0cc990 con 0x7fe18c15a5f0 2023-11-08T08:18:28.429 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe1925e4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe18c0b2ee0 0x7fe18c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:28.429 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe1915e2700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe18c0cc200 0x7fe18c0b27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40440/0 (socket says 172.21.15.116:40440) 2023-11-08T08:18:28.430 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe1915e2700 1 -- 172.21.15.116:0/2536697504 learned_addr learned my addr 172.21.15.116:0/2536697504 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:28.430 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe191de3700 1 --2- 172.21.15.116:0/2536697504 >> v2:172.21.15.129:3300/0 conn(0x7fe18c15a5f0 0x7fe18c0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:28.430 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe1925e4700 1 -- 172.21.15.116:0/2536697504 >> v2:172.21.15.129:3300/0 conn(0x7fe18c15a5f0 msgr2=0x7fe18c0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:28.430 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.426+0000 7fe1925e4700 1 --2- 172.21.15.116:0/2536697504 >> v2:172.21.15.129:3300/0 conn(0x7fe18c15a5f0 0x7fe18c0d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.430 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe1925e4700 1 -- 172.21.15.116:0/2536697504 >> v1:172.21.15.116:6789/0 conn(0x7fe18c0cc200 legacy=0x7fe18c0b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:28.431 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe1925e4700 1 -- 172.21.15.116:0/2536697504 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c0ccc70 con 0x7fe18c0b2ee0 2023-11-08T08:18:28.431 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe190de1700 1 -- 172.21.15.116:0/2536697504 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118361566 0 0) 0x7fe18c15a9c0 con 0x7fe18c0cc200 2023-11-08T08:18:28.431 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe191de3700 1 --2- 172.21.15.116:0/2536697504 >> v2:172.21.15.129:3300/0 conn(0x7fe18c15a5f0 0x7fe18c0d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:28.431 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe1925e4700 1 --2- 172.21.15.116:0/2536697504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ced30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0d1d40 tx=0x7fe18400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa55f89d3b3a610 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:28.432 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe190de1700 1 -- 172.21.15.116:0/2536697504 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18400e070 con 0x7fe18c0b2ee0 2023-11-08T08:18:28.432 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe190de1700 1 -- 172.21.15.116:0/2536697504 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18400bbd0 con 0x7fe18c0b2ee0 2023-11-08T08:18:28.432 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe190de1700 1 -- 172.21.15.116:0/2536697504 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184012430 con 0x7fe18c0b2ee0 2023-11-08T08:18:28.435 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe194047700 1 -- 172.21.15.116:0/2536697504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0b2ee0 msgr2=0x7fe18c0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:28.435 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.427+0000 7fe194047700 1 --2- 172.21.15.116:0/2536697504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ced30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0d1d40 tx=0x7fe18400b330 comp rx=0 tx=0).stop 2023-11-08T08:18:28.435 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.428+0000 7fe194047700 1 -- 172.21.15.116:0/2536697504 shutdown_connections 2023-11-08T08:18:28.436 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.428+0000 7fe194047700 1 --2- 172.21.15.116:0/2536697504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ced30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.436 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.428+0000 7fe194047700 1 --2- 172.21.15.116:0/2536697504 >> v2:172.21.15.129:3300/0 conn(0x7fe18c15a5f0 0x7fe18c0d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.436 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.428+0000 7fe194047700 1 -- 172.21.15.116:0/2536697504 >> 172.21.15.116:0/2536697504 conn(0x7fe18c06ea80 msgr2=0x7fe18c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:28.436 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.428+0000 7fe194047700 1 -- 172.21.15.116:0/2536697504 shutdown_connections 2023-11-08T08:18:28.437 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.428+0000 7fe194047700 1 -- 172.21.15.116:0/2536697504 wait complete. 2023-11-08T08:18:28.437 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.429+0000 7fe194047700 1 Processor -- start 2023-11-08T08:18:28.437 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.429+0000 7fe194047700 1 -- start start 2023-11-08T08:18:28.437 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:28.437 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 0x7fe18c14d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:28.438 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe18c15a5f0 0x7fe18c0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:28.438 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c1e5ca0 con 0x7fe18c0cc200 2023-11-08T08:18:28.438 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c1e5090 con 0x7fe18c0b2ee0 2023-11-08T08:18:28.438 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c1e5370 con 0x7fe18c15a5f0 2023-11-08T08:18:28.438 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 0x7fe18c14d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:28.439 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 0x7fe18c14d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40678/0 (socket says 172.21.15.116:40678) 2023-11-08T08:18:28.439 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 -- 172.21.15.116:0/3289986472 learned_addr learned my addr 172.21.15.116:0/3289986472 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:28.439 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe191de3700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:28.439 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1925e4700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe18c15a5f0 0x7fe18c0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:28.440 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 -- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe18c15a5f0 msgr2=0x7fe18c0b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:28.440 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe18c15a5f0 0x7fe18c0b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.440 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 -- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe18c0b2ee0 msgr2=0x7fe18c0ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:28.440 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.440 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 -- 172.21.15.116:0/3289986472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe184009a70 con 0x7fe18c0cc200 2023-11-08T08:18:28.441 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1915e2700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 0x7fe18c14d3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0011e0 tx=0x7fe17c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:28.441 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c004c80 con 0x7fe18c0cc200 2023-11-08T08:18:28.441 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c004e00 con 0x7fe18c0cc200 2023-11-08T08:18:28.441 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c0cd230 con 0x7fe18c0cc200 2023-11-08T08:18:28.441 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c007310 con 0x7fe18c0cc200 2023-11-08T08:18:28.442 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.430+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c0ccf50 con 0x7fe18c0cc200 2023-11-08T08:18:28.442 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.431+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fe17c01a020 con 0x7fe18c0cc200 2023-11-08T08:18:28.442 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.431+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe17c01d3f0 con 0x7fe18c0cc200 2023-11-08T08:18:28.442 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.431+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c0cd510 con 0x7fe18c0cc200 2023-11-08T08:18:28.443 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:28.434+0000 7f0604c66240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-08T08:18:28.443 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.435+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fe17c01d5b0 con 0x7fe18c0cc200 2023-11-08T08:18:28.504 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.503+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe18c1e5e10 con 0x7fe18c0cc200 2023-11-08T08:18:28.505 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.504+0000 7fe1827fc700 1 -- 172.21.15.116:0/3289986472 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe17c007520 con 0x7fe18c0cc200 2023-11-08T08:18:28.505 INFO:teuthology.misc.health.smithi116.stdout: 2023-11-08T08:18:28.505 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":10,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:27.217106+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:28.506 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 msgr2=0x7fe18c14d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:28.507 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 0x7fe18c14d3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0011e0 tx=0x7fe17c00ca30 comp rx=0 tx=0).stop 2023-11-08T08:18:28.507 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 shutdown_connections 2023-11-08T08:18:28.507 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe18c15a5f0 0x7fe18c0b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.507 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe18c0b2ee0 0x7fe18c0ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.508 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 --2- 172.21.15.116:0/3289986472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c0cc200 0x7fe18c14d3b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:28.508 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 >> 172.21.15.116:0/3289986472 conn(0x7fe18c06ea80 msgr2=0x7fe18c155d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:28.508 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 shutdown_connections 2023-11-08T08:18:28.508 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:28.505+0000 7fe194047700 1 -- 172.21.15.116:0/3289986472 wait complete. 2023-11-08T08:18:28.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-08T08:18:28.744 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:28.743+0000 7f8c088cb240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-08T08:18:28.891 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:28.890+0000 7f8c088cb240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-08T08:18:28.936 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:28.935+0000 7f0604c66240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-08T08:18:29.085 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:29.084+0000 7f0604c66240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-08T08:18:29.195 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:29.194+0000 7f8c088cb240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-08T08:18:29.395 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:29.394+0000 7f0604c66240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-08T08:18:29.721 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:29.720+0000 7f8c088cb240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-08T08:18:29.868 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:29.867+0000 7f8c088cb240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-08T08:18:29.924 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:29.923+0000 7f0604c66240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-08T08:18:30.071 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:30.070+0000 7f0604c66240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-08T08:18:30.223 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:30.222+0000 7f8c088cb240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-08T08:18:30.331 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:30.330+0000 7f8c088cb240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-08T08:18:30.452 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:30.451+0000 7f0604c66240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-08T08:18:30.551 INFO:tasks.ceph.mgr.x.smithi129.stderr:2023-11-08T08:18:30.550+0000 7f0604c66240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-08T08:18:31.344 INFO:tasks.ceph.osd.4.smithi129.stderr:2023-11-08T08:18:31.342+0000 7f5b935cf680 -1 osd.4 0 log_to_monitors true 2023-11-08T08:18:31.348 INFO:tasks.ceph.osd.7.smithi129.stderr:2023-11-08T08:18:31.347+0000 7fec40365680 -1 osd.7 0 log_to_monitors true 2023-11-08T08:18:31.362 INFO:tasks.ceph.osd.6.smithi129.stderr:2023-11-08T08:18:31.361+0000 7f4313eb1680 -1 osd.6 0 log_to_monitors true 2023-11-08T08:18:31.366 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:31.365+0000 7f8bc9cea700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-08T08:18:31.367 INFO:tasks.ceph.osd.5.smithi129.stderr:2023-11-08T08:18:31.366+0000 7fa1fc655680 -1 osd.5 0 log_to_monitors true 2023-11-08T08:18:31.371 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-08T08:18:31.370+0000 7f8bc9cea700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-08T08:18:31.371 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-08T08:18:31.370+0000 7f5158a94680 -1 osd.1 0 log_to_monitors true 2023-11-08T08:18:31.372 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-08T08:18:31.367+0000 7fa114f9f680 -1 osd.3 0 log_to_monitors true 2023-11-08T08:18:31.387 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-08T08:18:31.386+0000 7f0270960680 -1 osd.2 0 log_to_monitors true 2023-11-08T08:18:31.396 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-08T08:18:31.395+0000 7f14be7d5680 -1 osd.0 0 log_to_monitors true 2023-11-08T08:18:34.416 INFO:tasks.ceph.osd.4.smithi129.stderr:2023-11-08T08:18:34.415+0000 7f5b8716f700 -1 osd.4 0 waiting for initial osdmap 2023-11-08T08:18:34.441 INFO:tasks.ceph.osd.4.smithi129.stderr:2023-11-08T08:18:34.440+0000 7f5b7f54a700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.445 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-08T08:18:34.445+0000 7fa108b3f700 -1 osd.3 0 waiting for initial osdmap 2023-11-08T08:18:34.452 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-08T08:18:34.452+0000 7f514c634700 -1 osd.1 0 waiting for initial osdmap 2023-11-08T08:18:34.459 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-08T08:18:34.459+0000 7fa100f1a700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.464 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-08T08:18:34.463+0000 7f14b2375700 -1 osd.0 0 waiting for initial osdmap 2023-11-08T08:18:34.465 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-08T08:18:34.464+0000 7f5144a0f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.466 INFO:tasks.ceph.osd.7.smithi129.stderr:2023-11-08T08:18:34.466+0000 7fec33f05700 -1 osd.7 0 waiting for initial osdmap 2023-11-08T08:18:34.471 INFO:tasks.ceph.osd.7.smithi129.stderr:2023-11-08T08:18:34.470+0000 7fec2c2e0700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.473 INFO:tasks.ceph.osd.5.smithi129.stderr:2023-11-08T08:18:34.472+0000 7fa1f19f8700 -1 osd.5 0 waiting for initial osdmap 2023-11-08T08:18:34.473 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-08T08:18:34.472+0000 7f14aa750700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.492 INFO:tasks.ceph.osd.5.smithi129.stderr:2023-11-08T08:18:34.491+0000 7fa1e85d0700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.497 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-08T08:18:34.496+0000 7f0264500700 -1 osd.2 0 waiting for initial osdmap 2023-11-08T08:18:34.513 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-08T08:18:34.512+0000 7f025c8db700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:34.513 INFO:tasks.ceph.osd.6.smithi129.stderr:2023-11-08T08:18:34.512+0000 7f4307a51700 -1 osd.6 0 waiting for initial osdmap 2023-11-08T08:18:34.525 INFO:tasks.ceph.osd.6.smithi129.stderr:2023-11-08T08:18:34.524+0000 7f42ffe2c700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T08:18:35.323 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:35.448 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.447+0000 7f3e2b5d2700 1 Processor -- start 2023-11-08T08:18:35.449 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.447+0000 7f3e2b5d2700 1 -- start start 2023-11-08T08:18:35.450 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.448+0000 7f3e2b5d2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e2415a4d0 0x7f3e2406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.450 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.448+0000 7f3e2b5d2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e240cdf20 0x7f3e240cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.451 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.448+0000 7f3e2b5d2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2415e0d0 con 0x7f3e2406a7b0 2023-11-08T08:18:35.451 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.448+0000 7f3e2b5d2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2406ac30 con 0x7f3e240cdf20 2023-11-08T08:18:35.451 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.448+0000 7f3e2b5d2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e240ce310 con 0x7f3e2415a4d0 2023-11-08T08:18:35.451 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e240cdf20 0x7f3e240cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.451 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e240cdf20 0x7f3e240cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52020/0 (socket says 172.21.15.116:52020) 2023-11-08T08:18:35.452 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 -- 172.21.15.116:0/3899007337 learned_addr learned my addr 172.21.15.116:0/3899007337 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:35.452 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e2936e700 1 --2- 172.21.15.116:0/3899007337 >> v2:172.21.15.129:3300/0 conn(0x7f3e2415a4d0 0x7f3e2406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.452 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 -- 172.21.15.116:0/3899007337 >> v2:172.21.15.129:3300/0 conn(0x7f3e2415a4d0 msgr2=0x7f3e2406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:35.452 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 --2- 172.21.15.116:0/3899007337 >> v2:172.21.15.129:3300/0 conn(0x7f3e2415a4d0 0x7f3e2406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.452 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 -- 172.21.15.116:0/3899007337 >> v1:172.21.15.129:6789/0 conn(0x7f3e2406a7b0 legacy=0x7f3e240cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:35.453 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 -- 172.21.15.116:0/3899007337 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240ce560 con 0x7f3e240cdf20 2023-11-08T08:18:35.453 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.449+0000 7f3e29b6f700 1 --2- 172.21.15.116:0/3899007337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e240cb010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e2406b140 tx=0x7f3e14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e09f168a56a9ef2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:35.453 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.453+0000 7f3e23fff700 1 -- 172.21.15.116:0/3899007337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14004230 con 0x7f3e240cdf20 2023-11-08T08:18:35.454 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.453+0000 7f3e23fff700 1 -- 172.21.15.116:0/3899007337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e14004390 con 0x7f3e240cdf20 2023-11-08T08:18:35.454 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.453+0000 7f3e23fff700 1 -- 172.21.15.116:0/3899007337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e140036a0 con 0x7f3e240cdf20 2023-11-08T08:18:35.455 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/3899007337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 msgr2=0x7f3e240cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.455 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/3899007337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e240cb010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e2406b140 tx=0x7f3e14009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.456 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/3899007337 shutdown_connections 2023-11-08T08:18:35.456 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/3899007337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e240cb010 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.456 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/3899007337 >> v2:172.21.15.129:3300/0 conn(0x7f3e2415a4d0 0x7f3e2406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.456 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/3899007337 >> 172.21.15.116:0/3899007337 conn(0x7f3e2406c010 msgr2=0x7f3e240cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:35.457 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.454+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/3899007337 shutdown_connections 2023-11-08T08:18:35.457 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.455+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/3899007337 wait complete. 2023-11-08T08:18:35.457 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.456+0000 7f3e2b5d2700 1 Processor -- start 2023-11-08T08:18:35.458 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.457+0000 7f3e2b5d2700 1 -- start start 2023-11-08T08:18:35.458 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2b5d2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e2406a7b0 0x7f3e240c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.459 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2b5d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e2415c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.459 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2b5d2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 0x7f3e2414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.459 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2b5d2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2415d020 con 0x7f3e240cdf20 2023-11-08T08:18:35.459 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2b5d2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2415d300 con 0x7f3e2415a4d0 2023-11-08T08:18:35.459 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2b5d2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2415d5e0 con 0x7f3e2406a7b0 2023-11-08T08:18:35.461 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e29b6f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 0x7f3e2414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.461 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e29b6f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 0x7f3e2414ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:39268/0 (socket says 172.21.15.116:39268) 2023-11-08T08:18:35.461 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e29b6f700 1 -- 172.21.15.116:0/1820781669 learned_addr learned my addr 172.21.15.116:0/1820781669 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:35.461 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e28b6d700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e2415c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.462 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.458+0000 7f3e2936e700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e2406a7b0 0x7f3e240c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.462 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e28b6d700 1 -- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e2406a7b0 msgr2=0x7f3e240c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.462 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e28b6d700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e2406a7b0 0x7f3e240c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.462 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e28b6d700 1 -- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 msgr2=0x7f3e2414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.462 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e28b6d700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 0x7f3e2414ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.463 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e28b6d700 1 -- 172.21.15.116:0/1820781669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14009550 con 0x7f3e240cdf20 2023-11-08T08:18:35.463 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e29b6f700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 0x7f3e2414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:35.463 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e28b6d700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e2415c5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3e18001b60 tx=0x7f3e18001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:35.463 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.459+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180054e0 con 0x7f3e240cdf20 2023-11-08T08:18:35.463 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.460+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e180138e0 con 0x7f3e240cdf20 2023-11-08T08:18:35.464 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.460+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18013c90 con 0x7f3e240cdf20 2023-11-08T08:18:35.464 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.460+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2415e600 con 0x7f3e240cdf20 2023-11-08T08:18:35.464 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.460+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2415e320 con 0x7f3e240cdf20 2023-11-08T08:18:35.465 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.461+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2415e8e0 con 0x7f3e240cdf20 2023-11-08T08:18:35.465 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.461+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f3e1800a690 con 0x7f3e240cdf20 2023-11-08T08:18:35.465 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.462+0000 7f3e21ffb700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3e100682a0 0x7f3e1006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.466 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.462+0000 7f3e2936e700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3e100682a0 0x7f3e1006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.466 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.462+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f3e1808d2b0 con 0x7f3e240cdf20 2023-11-08T08:18:35.466 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.462+0000 7f3e2936e700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3e100682a0 0x7f3e1006a750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c011340 tx=0x7f3e1c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:35.467 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.467+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3e1805a190 con 0x7f3e240cdf20 2023-11-08T08:18:35.650 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.649+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e240af610 con 0x7f3e240cdf20 2023-11-08T08:18:35.650 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.649+0000 7f3e21ffb700 1 -- 172.21.15.116:0/1820781669 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10584 (secure 0 0 0) 0x7f3e240af610 con 0x7f3e240cdf20 2023-11-08T08:18:35.651 INFO:teuthology.misc.health.smithi116.stdout: 2023-11-08T08:18:35.651 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":13,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:35.447743+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:35.653 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.651+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3e100682a0 msgr2=0x7f3e1006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.653 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.651+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3e100682a0 0x7f3e1006a750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c011340 tx=0x7f3e1c00b040 comp rx=0 tx=0).stop 2023-11-08T08:18:35.653 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.651+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 msgr2=0x7f3e2415c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.654 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.651+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e2415c5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3e18001b60 tx=0x7f3e18001e70 comp rx=0 tx=0).stop 2023-11-08T08:18:35.654 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 shutdown_connections 2023-11-08T08:18:35.654 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3e100682a0 0x7f3e1006a750 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.654 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e2415a4d0 0x7f3e2414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.654 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e2406a7b0 0x7f3e240c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.655 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 --2- 172.21.15.116:0/1820781669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e240cdf20 0x7f3e2415c5d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.655 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 >> 172.21.15.116:0/1820781669 conn(0x7f3e2406c010 msgr2=0x7f3e240cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:35.655 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 shutdown_connections 2023-11-08T08:18:35.655 INFO:teuthology.misc.health.smithi116.stderr:2023-11-08T08:18:35.652+0000 7f3e2b5d2700 1 -- 172.21.15.116:0/1820781669 wait complete. 2023-11-08T08:18:35.664 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-08T08:18:35.664 INFO:tasks.ceph:Creating RBD pool 2023-11-08T08:18:35.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-08T08:18:35.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.785+0000 7f4afae72700 1 Processor -- start 2023-11-08T08:18:35.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.785+0000 7f4afae72700 1 -- start start 2023-11-08T08:18:35.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4afae72700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4afae72700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4af406a270 0x7f4af406a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4afae72700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af406ac10 con 0x7f4af40718e0 2023-11-08T08:18:35.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4afae72700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af415cbe0 con 0x7f4af406a270 2023-11-08T08:18:35.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4afae72700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41607a0 con 0x7f4af415c810 2023-11-08T08:18:35.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4af406a270 0x7f4af406a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4af406a270 0x7f4af406a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52034/0 (socket says 172.21.15.116:52034) 2023-11-08T08:18:35.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af3fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4af40718e0 0x7f4af406c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58752/0 (socket says 172.21.15.116:58752) 2023-11-08T08:18:35.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 -- 172.21.15.116:0/3119976807 learned_addr learned my addr 172.21.15.116:0/3119976807 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:35.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af8c0e700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 -- 172.21.15.116:0/3119976807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 msgr2=0x7f4af4160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:35.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 -- 172.21.15.116:0/3119976807 >> v1:172.21.15.116:6789/0 conn(0x7f4af40718e0 legacy=0x7f4af406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 -- 172.21.15.116:0/3119976807 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4160a50 con 0x7f4af406a270 2023-11-08T08:18:35.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af37fe700 1 -- 172.21.15.116:0/3119976807 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913253351 0 0) 0x7f4af406ac10 con 0x7f4af40718e0 2023-11-08T08:18:35.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af8c0e700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:35.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af940f700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af406a6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009e50 tx=0x7f4aec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc23abeaa388c97 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:35.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.786+0000 7f4af37fe700 1 -- 172.21.15.116:0/3119976807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec00e070 con 0x7f4af406a270 2023-11-08T08:18:35.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4af37fe700 1 -- 172.21.15.116:0/3119976807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aec00b9c0 con 0x7f4af406a270 2023-11-08T08:18:35.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4af37fe700 1 -- 172.21.15.116:0/3119976807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec00bd70 con 0x7f4af406a270 2023-11-08T08:18:35.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 -- 172.21.15.116:0/3119976807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 msgr2=0x7f4af406a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af406a6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009e50 tx=0x7f4aec00b330 comp rx=0 tx=0).stop 2023-11-08T08:18:35.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 -- 172.21.15.116:0/3119976807 shutdown_connections 2023-11-08T08:18:35.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af406a6d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 --2- 172.21.15.116:0/3119976807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 -- 172.21.15.116:0/3119976807 >> 172.21.15.116:0/3119976807 conn(0x7f4af406c010 msgr2=0x7f4af40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:35.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 -- 172.21.15.116:0/3119976807 shutdown_connections 2023-11-08T08:18:35.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.787+0000 7f4afae72700 1 -- 172.21.15.116:0/3119976807 wait complete. 2023-11-08T08:18:35.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.788+0000 7f4afae72700 1 Processor -- start 2023-11-08T08:18:35.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.788+0000 7f4afae72700 1 -- start start 2023-11-08T08:18:35.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4afae72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af4150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4afae72700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af40718e0 0x7f4af41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4afae72700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4afae72700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41507b0 con 0x7f4af406a270 2023-11-08T08:18:35.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4afae72700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4157bd0 con 0x7f4af40718e0 2023-11-08T08:18:35.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4afae72700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4157eb0 con 0x7f4af415c810 2023-11-08T08:18:35.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af4150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af4150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52042/0 (socket says 172.21.15.116:52042) 2023-11-08T08:18:35.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 -- 172.21.15.116:0/2539630185 learned_addr learned my addr 172.21.15.116:0/2539630185 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:35.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 -- 172.21.15.116:0/2539630185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 msgr2=0x7f4af4152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 -- 172.21.15.116:0/2539630185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af40718e0 msgr2=0x7f4af41529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af40718e0 0x7f4af41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.789+0000 7f4af8c0e700 1 -- 172.21.15.116:0/2539630185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec009b30 con 0x7f4af406a270 2023-11-08T08:18:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.790+0000 7f4af8c0e700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af4150270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c7f0 tx=0x7f4ae800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.790+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8011800 con 0x7f4af406a270 2023-11-08T08:18:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.790+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae8011960 con 0x7f4af406a270 2023-11-08T08:18:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.790+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af4161010 con 0x7f4af406a270 2023-11-08T08:18:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.790+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8011d10 con 0x7f4af406a270 2023-11-08T08:18:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.790+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af4160d30 con 0x7f4af406a270 2023-11-08T08:18:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.791+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af40ceb20 con 0x7f4af406a270 2023-11-08T08:18:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.792+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f4ae8021070 con 0x7f4af406a270 2023-11-08T08:18:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.792+0000 7f4af17fa700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ad80682a0 0x7f4ad806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.792+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f4ae8085880 con 0x7f4af406a270 2023-11-08T08:18:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.792+0000 7f4af3fff700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ad80682a0 0x7f4ad806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.792+0000 7f4af3fff700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ad80682a0 0x7f4ad806a750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00011e0 tx=0x7f4ae00079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.796+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ae8056230 con 0x7f4af406a270 2023-11-08T08:18:35.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:35.977+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f4af40743f0 con 0x7f4af406a270 2023-11-08T08:18:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.454+0000 7f4af17fa700 1 -- 172.21.15.116:0/2539630185 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f4ae8055e20 con 0x7f4af406a270 2023-11-08T08:18:36.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ad80682a0 msgr2=0x7f4ad806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ad80682a0 0x7f4ad806a750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00011e0 tx=0x7f4ae00079b0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 msgr2=0x7f4af4150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af4150270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c7f0 tx=0x7f4ae800cbb0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 shutdown_connections 2023-11-08T08:18:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4af415c810 0x7f4af4152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af406a270 0x7f4af4150270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ad80682a0 0x7f4ad806a750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 --2- 172.21.15.116:0/2539630185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af40718e0 0x7f4af41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 >> 172.21.15.116:0/2539630185 conn(0x7f4af406c010 msgr2=0x7f4af4154500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:36.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 shutdown_connections 2023-11-08T08:18:36.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.457+0000 7f4afae72700 1 -- 172.21.15.116:0/2539630185 wait complete. 2023-11-08T08:18:36.461 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'rbd' created 2023-11-08T08:18:36.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-08T08:18:36.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.593+0000 7f534ba6a700 1 Processor -- start 2023-11-08T08:18:36.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.593+0000 7f534ba6a700 1 -- start start 2023-11-08T08:18:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f534ba6a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53440cb4b0 0x7f5344159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f534ba6a700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440ceba0 con 0x7f534406c390 2023-11-08T08:18:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f534ba6a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440c9b40 con 0x7f53440ce7d0 2023-11-08T08:18:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f534ba6a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53440ca0e0 con 0x7f53440cb4b0 2023-11-08T08:18:36.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5349005700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53440cb4b0 0x7f5344159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:36.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5349005700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53440cb4b0 0x7f5344159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52056/0 (socket says 172.21.15.116:52056) 2023-11-08T08:18:36.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f534a007700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f534406c390 0x7f534406a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58762/0 (socket says 172.21.15.116:58762) 2023-11-08T08:18:36.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5349005700 1 -- 172.21.15.116:0/3584148615 learned_addr learned my addr 172.21.15.116:0/3584148615 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5349005700 1 -- 172.21.15.116:0/3584148615 >> v1:172.21.15.129:6789/0 conn(0x7f53440ce7d0 legacy=0x7f5344070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5349005700 1 -- 172.21.15.116:0/3584148615 >> v1:172.21.15.116:6789/0 conn(0x7f534406c390 legacy=0x7f534406a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5349005700 1 -- 172.21.15.116:0/3584148615 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53440cb950 con 0x7f53440cb4b0 2023-11-08T08:18:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.594+0000 7f5348804700 1 -- 172.21.15.116:0/3584148615 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552054213 0 0) 0x7f53440ceba0 con 0x7f534406c390 2023-11-08T08:18:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f5349005700 1 --2- 172.21.15.116:0/3584148615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440cb4b0 0x7f5344159ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f53400098f0 tx=0x7f5340009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcaaa3cc5b2be7c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f5348804700 1 -- 172.21.15.116:0/3584148615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534000b8a0 con 0x7f53440cb4b0 2023-11-08T08:18:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f5348804700 1 -- 172.21.15.116:0/3584148615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534000ba00 con 0x7f53440cb4b0 2023-11-08T08:18:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f5348804700 1 -- 172.21.15.116:0/3584148615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534000bd90 con 0x7f53440cb4b0 2023-11-08T08:18:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 -- 172.21.15.116:0/3584148615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440cb4b0 msgr2=0x7f5344159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3584148615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440cb4b0 0x7f5344159ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f53400098f0 tx=0x7f5340009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 -- 172.21.15.116:0/3584148615 shutdown_connections 2023-11-08T08:18:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3584148615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440cb4b0 0x7f5344159ab0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 -- 172.21.15.116:0/3584148615 >> 172.21.15.116:0/3584148615 conn(0x7f53440cf370 msgr2=0x7f53440cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 -- 172.21.15.116:0/3584148615 shutdown_connections 2023-11-08T08:18:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.595+0000 7f534ba6a700 1 -- 172.21.15.116:0/3584148615 wait complete. 2023-11-08T08:18:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.596+0000 7f534ba6a700 1 Processor -- start 2023-11-08T08:18:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.596+0000 7f534ba6a700 1 -- start start 2023-11-08T08:18:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534ba6a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f534406c390 0x7f53440c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534ba6a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53440cb4b0 0x7f5344159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534ba6a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 0x7f53440c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534ba6a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440c4450 con 0x7f53440ce7d0 2023-11-08T08:18:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534ba6a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440c3810 con 0x7f534406c390 2023-11-08T08:18:36.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534ba6a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53440c3ab0 con 0x7f53440cb4b0 2023-11-08T08:18:36.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534a007700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 0x7f53440c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:36.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534a007700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 0x7f53440c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52060/0 (socket says 172.21.15.116:52060) 2023-11-08T08:18:36.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534a007700 1 -- 172.21.15.116:0/3921443876 learned_addr learned my addr 172.21.15.116:0/3921443876 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f5349005700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53440cb4b0 0x7f5344159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.597+0000 7f534a007700 1 -- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53440cb4b0 msgr2=0x7f5344159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534a007700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53440cb4b0 0x7f5344159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534a007700 1 -- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f534406c390 msgr2=0x7f53440c4870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534a007700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f534406c390 0x7f53440c4870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:36.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534a007700 1 -- 172.21.15.116:0/3921443876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340009550 con 0x7f53440ce7d0 2023-11-08T08:18:36.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534a007700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 0x7f53440c8250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f533c00eeb0 tx=0x7f533c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:36.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c00cea0 con 0x7f53440ce7d0 2023-11-08T08:18:36.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533c0070d0 con 0x7f53440ce7d0 2023-11-08T08:18:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53440cef80 con 0x7f53440ce7d0 2023-11-08T08:18:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c007480 con 0x7f53440ce7d0 2023-11-08T08:18:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.598+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53440c9e00 con 0x7f53440ce7d0 2023-11-08T08:18:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.599+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53440cf110 con 0x7f53440ce7d0 2023-11-08T08:18:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.599+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f533c0075e0 con 0x7f53440ce7d0 2023-11-08T08:18:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.600+0000 7f533a7fc700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53280682a0 0x7f532806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:36.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.600+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 5188+0+0 (secure 0 0 0) 0x7f533c0887a0 con 0x7f53440ce7d0 2023-11-08T08:18:36.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.600+0000 7f5349806700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53280682a0 0x7f532806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:36.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.600+0000 7f5349806700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53280682a0 0x7f532806a750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5334000f60 tx=0x7f5334007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:36.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.604+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f533c0553b0 con 0x7f53440ce7d0 2023-11-08T08:18:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:36.789+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f53440743f0 con 0x7f53440ce7d0 2023-11-08T08:18:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.455+0000 7f533a7fc700 1 -- 172.21.15.116:0/3921443876 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f533c0150c0 con 0x7f53440ce7d0 2023-11-08T08:18:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53280682a0 msgr2=0x7f532806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53280682a0 0x7f532806a750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5334000f60 tx=0x7f5334007a40 comp rx=0 tx=0).stop 2023-11-08T08:18:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 msgr2=0x7f53440c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 0x7f53440c8250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f533c00eeb0 tx=0x7f533c00ef90 comp rx=0 tx=0).stop 2023-11-08T08:18:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 shutdown_connections 2023-11-08T08:18:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53440ce7d0 0x7f53440c8250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53280682a0 0x7f532806a750 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f534406c390 0x7f53440c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.458+0000 7f534ba6a700 1 --2- 172.21.15.116:0/3921443876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53440cb4b0 0x7f5344159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.459+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 >> 172.21.15.116:0/3921443876 conn(0x7f53440cf370 msgr2=0x7f53440cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.459+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 shutdown_connections 2023-11-08T08:18:37.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.459+0000 7f534ba6a700 1 -- 172.21.15.116:0/3921443876 wait complete. 2023-11-08T08:18:37.464 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-08T08:18:37.476 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-08T08:18:37.477 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-08T08:18:37.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-08T08:18:37.484 INFO:tasks.ceph.mds.a:Started 2023-11-08T08:18:37.485 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-08T08:18:37.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-08T08:18:37.493 INFO:tasks.ceph.mds.c:Started 2023-11-08T08:18:37.493 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-08T08:18:37.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-08T08:18:37.497 INFO:tasks.ceph.mds.b:Started 2023-11-08T08:18:37.497 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-08T08:18:37.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-08T08:18:37.504 INFO:tasks.ceph.mds.d:Started 2023-11-08T08:18:37.505 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-08T08:18:37.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:37.574 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-08T08:18:37.580 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-08T08:18:37.583 INFO:tasks.ceph.mds.b.smithi129.stdout:starting mds.b at 2023-11-08T08:18:37.591 INFO:tasks.ceph.mds.d.smithi129.stdout:starting mds.d at 2023-11-08T08:18:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.647+0000 7f08e6606700 1 Processor -- start 2023-11-08T08:18:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.647+0000 7f08e6606700 1 -- start start 2023-11-08T08:18:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.647+0000 7f08e6606700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e015b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08e6606700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e015af30 con 0x7f08e006e400 2023-11-08T08:18:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08e6606700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e015ef40 con 0x7f08e015e010 2023-11-08T08:18:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08e6606700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e015a9d0 con 0x7f08e00d15f0 2023-11-08T08:18:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08dffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f08e006e400 0x7f08e006e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58802/0 (socket says 172.21.15.116:58802) 2023-11-08T08:18:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08dffff700 1 -- 172.21.15.116:0/2714348709 learned_addr learned my addr 172.21.15.116:0/2714348709 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08df7fe700 1 --2- 172.21.15.116:0/2714348709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e015b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812962673 0 0) 0x7f08e015af30 con 0x7f08e006e400 2023-11-08T08:18:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c80035b0 con 0x7f08e006e400 2023-11-08T08:18:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592160776 0 0) 0x7f08e015ef40 con 0x7f08e015e010 2023-11-08T08:18:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e015af30 con 0x7f08e015e010 2023-11-08T08:18:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08df7fe700 1 -- 172.21.15.116:0/2714348709 >> v1:172.21.15.129:6789/0 conn(0x7f08e015e010 legacy=0x7f08e00ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08df7fe700 1 -- 172.21.15.116:0/2714348709 >> v1:172.21.15.116:6789/0 conn(0x7f08e006e400 legacy=0x7f08e006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08df7fe700 1 -- 172.21.15.116:0/2714348709 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e015e980 con 0x7f08e00d15f0 2023-11-08T08:18:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08df7fe700 1 --2- 172.21.15.116:0/2714348709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e015b3c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009870 tx=0x7f08d4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=747f50fd5340e185 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4011830 con 0x7f08e00d15f0 2023-11-08T08:18:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4011990 con 0x7f08e00d15f0 2023-11-08T08:18:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.648+0000 7f08deffd700 1 -- 172.21.15.116:0/2714348709 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4011d20 con 0x7f08e00d15f0 2023-11-08T08:18:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 -- 172.21.15.116:0/2714348709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 msgr2=0x7f08e015b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 --2- 172.21.15.116:0/2714348709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e015b3c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009870 tx=0x7f08d4009b80 comp rx=0 tx=0).stop 2023-11-08T08:18:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 -- 172.21.15.116:0/2714348709 shutdown_connections 2023-11-08T08:18:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 --2- 172.21.15.116:0/2714348709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e015b3c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 -- 172.21.15.116:0/2714348709 >> 172.21.15.116:0/2714348709 conn(0x7f08e006a270 msgr2=0x7f08e006a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 -- 172.21.15.116:0/2714348709 shutdown_connections 2023-11-08T08:18:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.649+0000 7f08e6606700 1 -- 172.21.15.116:0/2714348709 wait complete. 2023-11-08T08:18:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.650+0000 7f08e6606700 1 Processor -- start 2023-11-08T08:18:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.650+0000 7f08e6606700 1 -- start start 2023-11-08T08:18:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e6606700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08e006e400 0x7f08e00b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e6606700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e0152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e6606700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 0x7f08e00cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e6606700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b1e80 con 0x7f08e015e010 2023-11-08T08:18:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e6606700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b2160 con 0x7f08e006e400 2023-11-08T08:18:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e6606700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b2440 con 0x7f08e00d15f0 2023-11-08T08:18:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 0x7f08e00cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 0x7f08e00cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52114/0 (socket says 172.21.15.116:52114) 2023-11-08T08:18:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 -- 172.21.15.116:0/1029799276 learned_addr learned my addr 172.21.15.116:0/1029799276 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08df7fe700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e0152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08dffff700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08e006e400 0x7f08e00b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 -- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 msgr2=0x7f08e0152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e0152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 -- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08e006e400 msgr2=0x7f08e00b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08e006e400 0x7f08e00b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 -- 172.21.15.116:0/1029799276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d40094a0 con 0x7f08e015e010 2023-11-08T08:18:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.651+0000 7f08e4ba3700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 0x7f08e00cba60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08d800cbc0 tx=0x7f08d800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:37.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.652+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8003c00 con 0x7f08e015e010 2023-11-08T08:18:37.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.652+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8003d60 con 0x7f08e015e010 2023-11-08T08:18:37.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.652+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e015abc0 con 0x7f08e015e010 2023-11-08T08:18:37.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.652+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d800aaa0 con 0x7f08e015e010 2023-11-08T08:18:37.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.652+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e015ec60 con 0x7f08e015e010 2023-11-08T08:18:37.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.652+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e015b900 con 0x7f08e015e010 2023-11-08T08:18:37.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.653+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f08d8010050 con 0x7f08e015e010 2023-11-08T08:18:37.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.653+0000 7f08dcff9700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08cc0682a0 0x7f08cc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:37.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.654+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f08d8086190 con 0x7f08e015e010 2023-11-08T08:18:37.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.654+0000 7f08dffff700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08cc0682a0 0x7f08cc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:37.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.654+0000 7f08dffff700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08cc0682a0 0x7f08cc06a750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f08d0007fd0 tx=0x7f08d0007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:37.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.657+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08d8052e30 con 0x7f08e015e010 2023-11-08T08:18:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.857+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08e00743f0 con 0x7f08e015e010 2023-11-08T08:18:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.858+0000 7f08dcff9700 1 -- 172.21.15.116:0/1029799276 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+14281 (secure 0 0 0) 0x7f08d8056450 con 0x7f08e015e010 2023-11-08T08:18:37.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:37.859 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:37.450829+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08cc0682a0 msgr2=0x7f08cc06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08cc0682a0 0x7f08cc06a750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f08d0007fd0 tx=0x7f08d0007dc0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 msgr2=0x7f08e00cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 0x7f08e00cba60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08d800cbc0 tx=0x7f08d800ced0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 shutdown_connections 2023-11-08T08:18:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e015e010 0x7f08e00cba60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08cc0682a0 0x7f08cc06a750 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08e006e400 0x7f08e00b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 --2- 172.21.15.116:0/1029799276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08e00d15f0 0x7f08e0152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 >> 172.21.15.116:0/1029799276 conn(0x7f08e006a270 msgr2=0x7f08e0153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 shutdown_connections 2023-11-08T08:18:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:37.860+0000 7f08e6606700 1 -- 172.21.15.116:0/1029799276 wait complete. 2023-11-08T08:18:37.874 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:18:37.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:18:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.000+0000 7f2f16dda700 1 Processor -- start 2023-11-08T08:18:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.000+0000 7f2f16dda700 1 -- start start 2023-11-08T08:18:38.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f16dda700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2f1006b220 0x7f2f100d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f16dda700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100d2d30 con 0x7f2f10160cb0 2023-11-08T08:18:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f16dda700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100d0b40 con 0x7f2f1015c8c0 2023-11-08T08:18:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f16dda700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f100d2a50 con 0x7f2f1006b220 2023-11-08T08:18:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0ffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2f10160cb0 0x7f2f1006ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58812/0 (socket says 172.21.15.116:58812) 2023-11-08T08:18:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0ffff700 1 -- 172.21.15.116:0/3390571932 learned_addr learned my addr 172.21.15.116:0/3390571932 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f15377700 1 --2- 172.21.15.116:0/3390571932 >> v2:172.21.15.129:3300/0 conn(0x7f2f1006b220 0x7f2f100d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23193844 0 0) 0x7f2efc001110 con 0x7f2f10160cb0 2023-11-08T08:18:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f000035b0 con 0x7f2f10160cb0 2023-11-08T08:18:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852667479 0 0) 0x7f2f000035b0 con 0x7f2f10160cb0 2023-11-08T08:18:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 >> v2:172.21.15.129:3300/0 conn(0x7f2f1006b220 msgr2=0x7f2f100d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 --2- 172.21.15.116:0/3390571932 >> v2:172.21.15.129:3300/0 conn(0x7f2f1006b220 0x7f2f100d0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 >> v1:172.21.15.129:6789/0 conn(0x7f2f1015c8c0 legacy=0x7f2f100b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100cd760 con 0x7f2f10160cb0 2023-11-08T08:18:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f2efc001800 con 0x7f2f10160cb0 2023-11-08T08:18:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 >> v1:172.21.15.116:6789/0 conn(0x7f2f10160cb0 legacy=0x7f2f1006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00003a00 0x7f2f00003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f00004850 0x7f2f00004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f00005200 0x7f2f0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100d2a50 con 0x7f2f00004850 2023-11-08T08:18:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0000c060 con 0x7f2f00005200 2023-11-08T08:18:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0000c220 con 0x7f2f00003a00 2023-11-08T08:18:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.001+0000 7f2f0ffff700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f00004850 0x7f2f00004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0ffff700 1 -- 172.21.15.116:0/3390571932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00003a00 msgr2=0x7f2f00003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0ffff700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00003a00 0x7f2f00003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0ffff700 1 -- 172.21.15.116:0/3390571932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f00005200 msgr2=0x7f2f0000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0ffff700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f00005200 0x7f2f0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f15377700 1 -- 172.21.15.116:0/3390571932 reap_dead start 2023-11-08T08:18:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0ffff700 1 -- 172.21.15.116:0/3390571932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000035b0 con 0x7f2f00004850 2023-11-08T08:18:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0ffff700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f00004850 0x7f2f00004cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00c240 tx=0x7f2f00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f2cfbdc081b6f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00cde0 con 0x7f2f00004850 2023-11-08T08:18:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc005960 con 0x7f2f00004850 2023-11-08T08:18:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f0f7fe700 1 -- 172.21.15.116:0/3390571932 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc005cf0 con 0x7f2f00004850 2023-11-08T08:18:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f16dda700 1 -- 172.21.15.116:0/3390571932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f00004850 msgr2=0x7f2f00004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.002+0000 7f2f16dda700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f00004850 0x7f2f00004cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00c240 tx=0x7f2f00005880 comp rx=0 tx=0).stop 2023-11-08T08:18:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.003+0000 7f2f16dda700 1 -- 172.21.15.116:0/3390571932 shutdown_connections 2023-11-08T08:18:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.003+0000 7f2f16dda700 1 --2- 172.21.15.116:0/3390571932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f00004850 0x7f2f00004cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.003+0000 7f2f16dda700 1 -- 172.21.15.116:0/3390571932 >> 172.21.15.116:0/3390571932 conn(0x7f2f1006c520 msgr2=0x7f2f1006c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.003+0000 7f2f16dda700 1 -- 172.21.15.116:0/3390571932 shutdown_connections 2023-11-08T08:18:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.003+0000 7f2f16dda700 1 -- 172.21.15.116:0/3390571932 wait complete. 2023-11-08T08:18:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.004+0000 7f2f16dda700 1 Processor -- start 2023-11-08T08:18:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.004+0000 7f2f16dda700 1 -- start start 2023-11-08T08:18:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00004850 0x7f2f1015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1014d5d0 0x7f2f1015eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 0x7f2f100b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100b86f0 con 0x7f2f100b7d40 2023-11-08T08:18:38.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1015f410 con 0x7f2f1014d5d0 2023-11-08T08:18:38.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1014da40 con 0x7f2f00004850 2023-11-08T08:18:38.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 0x7f2f100b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 0x7f2f100b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52120/0 (socket says 172.21.15.116:52120) 2023-11-08T08:18:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 -- 172.21.15.116:0/1273641734 learned_addr learned my addr 172.21.15.116:0/1273641734 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f14b76700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00004850 0x7f2f1015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f0ffff700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1014d5d0 0x7f2f1015eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 -- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00004850 msgr2=0x7f2f1015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00004850 0x7f2f1015cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 -- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1014d5d0 msgr2=0x7f2f1015eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1014d5d0 0x7f2f1015eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 -- 172.21.15.116:0/1273641734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f10161120 con 0x7f2f100b7d40 2023-11-08T08:18:38.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f15377700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 0x7f2f100b81b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f100b8c50 tx=0x7f2f0800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08010040 con 0x7f2f100b7d40 2023-11-08T08:18:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0800b8c0 con 0x7f2f100b7d40 2023-11-08T08:18:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800bc70 con 0x7f2f100b7d40 2023-11-08T08:18:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f100cda40 con 0x7f2f100b7d40 2023-11-08T08:18:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.005+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100d2700 con 0x7f2f100b7d40 2023-11-08T08:18:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.006+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10067d70 con 0x7f2f100b7d40 2023-11-08T08:18:38.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.007+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f2f0801d400 con 0x7f2f100b7d40 2023-11-08T08:18:38.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.007+0000 7f2f0d7fa700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2ef40682a0 0x7f2ef406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.007+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2f08085e40 con 0x7f2f100b7d40 2023-11-08T08:18:38.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.007+0000 7f2f14b76700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2ef40682a0 0x7f2ef406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.007+0000 7f2f14b76700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2ef40682a0 0x7f2ef406a750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2f04000ff0 tx=0x7f2f04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.011+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f080529e0 con 0x7f2f100b7d40 2023-11-08T08:18:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.190+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2f100743f0 con 0x7f2f100b7d40 2023-11-08T08:18:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.191+0000 7f2f0d7fa700 1 -- 172.21.15.116:0/1273641734 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2f08056000 con 0x7f2f100b7d40 2023-11-08T08:18:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:18:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2ef40682a0 msgr2=0x7f2ef406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2ef40682a0 0x7f2ef406a750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2f04000ff0 tx=0x7f2f04009040 comp rx=0 tx=0).stop 2023-11-08T08:18:38.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 msgr2=0x7f2f100b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 0x7f2f100b81b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f100b8c50 tx=0x7f2f0800ed30 comp rx=0 tx=0).stop 2023-11-08T08:18:38.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 shutdown_connections 2023-11-08T08:18:38.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f100b7d40 0x7f2f100b81b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f00004850 0x7f2f1015cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2ef40682a0 0x7f2ef406a750 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 --2- 172.21.15.116:0/1273641734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1014d5d0 0x7f2f1015eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 >> 172.21.15.116:0/1273641734 conn(0x7f2f1006c520 msgr2=0x7f2f10157560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:38.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 shutdown_connections 2023-11-08T08:18:38.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.193+0000 7f2f16dda700 1 -- 172.21.15.116:0/1273641734 wait complete. 2023-11-08T08:18:38.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:18:38.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.332+0000 7f2e3aa65700 1 Processor -- start 2023-11-08T08:18:38.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.332+0000 7f2e3aa65700 1 -- start start 2023-11-08T08:18:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e3aa65700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e340cdf20 0x7f2e34165420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e3aa65700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3415e0d0 con 0x7f2e3406a980 2023-11-08T08:18:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e3aa65700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e340cc020 con 0x7f2e3415a4d0 2023-11-08T08:18:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e3aa65700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340ce380 con 0x7f2e340cdf20 2023-11-08T08:18:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e33fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2e3406a980 0x7f2e340cd810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58824/0 (socket says 172.21.15.116:58824) 2023-11-08T08:18:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e33fff700 1 -- 172.21.15.116:0/3341425848 learned_addr learned my addr 172.21.15.116:0/3341425848 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e39002700 1 --2- 172.21.15.116:0/3341425848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e340cdf20 0x7f2e34165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e337fe700 1 -- 172.21.15.116:0/3341425848 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592328487 0 0) 0x7f2e3415e0d0 con 0x7f2e3406a980 2023-11-08T08:18:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e337fe700 1 -- 172.21.15.116:0/3341425848 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e240035b0 con 0x7f2e3406a980 2023-11-08T08:18:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e39002700 1 -- 172.21.15.116:0/3341425848 >> v1:172.21.15.129:6789/0 conn(0x7f2e3415a4d0 legacy=0x7f2e3406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e337fe700 1 -- 172.21.15.116:0/3341425848 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 936358564 0 0) 0x7f2e240035b0 con 0x7f2e3406a980 2023-11-08T08:18:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e39002700 1 -- 172.21.15.116:0/3341425848 >> v1:172.21.15.116:6789/0 conn(0x7f2e3406a980 legacy=0x7f2e340cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.333+0000 7f2e39002700 1 -- 172.21.15.116:0/3341425848 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340ce560 con 0x7f2e340cdf20 2023-11-08T08:18:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e39002700 1 --2- 172.21.15.116:0/3341425848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e340cdf20 0x7f2e34165420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009dc0 tx=0x7f2e2c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=364c25e2296695f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e337fe700 1 -- 172.21.15.116:0/3341425848 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c00e070 con 0x7f2e340cdf20 2023-11-08T08:18:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e337fe700 1 -- 172.21.15.116:0/3341425848 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2c003a20 con 0x7f2e340cdf20 2023-11-08T08:18:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e337fe700 1 -- 172.21.15.116:0/3341425848 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20001670 con 0x7f2e340cdf20 2023-11-08T08:18:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 -- 172.21.15.116:0/3341425848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e340cdf20 msgr2=0x7f2e34165420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/3341425848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e340cdf20 0x7f2e34165420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009dc0 tx=0x7f2e2c00b330 comp rx=0 tx=0).stop 2023-11-08T08:18:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 -- 172.21.15.116:0/3341425848 shutdown_connections 2023-11-08T08:18:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/3341425848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e340cdf20 0x7f2e34165420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 -- 172.21.15.116:0/3341425848 >> 172.21.15.116:0/3341425848 conn(0x7f2e3406c010 msgr2=0x7f2e340cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 -- 172.21.15.116:0/3341425848 shutdown_connections 2023-11-08T08:18:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.334+0000 7f2e3aa65700 1 -- 172.21.15.116:0/3341425848 wait complete. 2023-11-08T08:18:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.335+0000 7f2e3aa65700 1 Processor -- start 2023-11-08T08:18:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 -- start start 2023-11-08T08:18:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e3406a980 0x7f2e3415d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e340cdf20 0x7f2e3415b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 0x7f2e34150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3415d920 con 0x7f2e3415a4d0 2023-11-08T08:18:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340b1180 con 0x7f2e340cdf20 2023-11-08T08:18:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.336+0000 7f2e3aa65700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340b1460 con 0x7f2e3406a980 2023-11-08T08:18:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 0x7f2e34150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 0x7f2e34150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52148/0 (socket says 172.21.15.116:52148) 2023-11-08T08:18:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 -- 172.21.15.116:0/910553566 learned_addr learned my addr 172.21.15.116:0/910553566 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e38801700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e3406a980 0x7f2e3415d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e33fff700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e340cdf20 0x7f2e3415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 -- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e3406a980 msgr2=0x7f2e3415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e3406a980 0x7f2e3415d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 -- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e340cdf20 msgr2=0x7f2e3415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e340cdf20 0x7f2e3415b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 -- 172.21.15.116:0/910553566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c009a70 con 0x7f2e3415a4d0 2023-11-08T08:18:38.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e39002700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 0x7f2e34150b70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00b370 tx=0x7f2e2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c004600 con 0x7f2e3415a4d0 2023-11-08T08:18:38.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2c004760 con 0x7f2e3415a4d0 2023-11-08T08:18:38.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c01d470 con 0x7f2e3415a4d0 2023-11-08T08:18:38.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3415e600 con 0x7f2e3415a4d0 2023-11-08T08:18:38.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.337+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3415e320 con 0x7f2e3415a4d0 2023-11-08T08:18:38.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.338+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3415e8e0 con 0x7f2e3415a4d0 2023-11-08T08:18:38.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.339+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f2e2c018030 con 0x7f2e3415a4d0 2023-11-08T08:18:38.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.339+0000 7f2e317fa700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e180682a0 0x7f2e1806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.339+0000 7f2e38801700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e180682a0 0x7f2e1806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.339+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2e2c086e60 con 0x7f2e3415a4d0 2023-11-08T08:18:38.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.340+0000 7f2e38801700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e180682a0 0x7f2e1806a750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007fd0 tx=0x7f2e28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.344+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e2c053a60 con 0x7f2e3415a4d0 2023-11-08T08:18:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.531+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e340743f0 con 0x7f2e3415a4d0 2023-11-08T08:18:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.531+0000 7f2e317fa700 1 -- 172.21.15.116:0/910553566 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e2c057080 con 0x7f2e3415a4d0 2023-11-08T08:18:38.533 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:18:38.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e180682a0 msgr2=0x7f2e1806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e180682a0 0x7f2e1806a750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007fd0 tx=0x7f2e28009040 comp rx=0 tx=0).stop 2023-11-08T08:18:38.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 msgr2=0x7f2e34150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 0x7f2e34150b70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00b370 tx=0x7f2e2c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 shutdown_connections 2023-11-08T08:18:38.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3415a4d0 0x7f2e34150b70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e3406a980 0x7f2e3415d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e180682a0 0x7f2e1806a750 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 --2- 172.21.15.116:0/910553566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e340cdf20 0x7f2e3415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 >> 172.21.15.116:0/910553566 conn(0x7f2e3406c010 msgr2=0x7f2e341524d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 shutdown_connections 2023-11-08T08:18:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.534+0000 7f2e3aa65700 1 -- 172.21.15.116:0/910553566 wait complete. 2023-11-08T08:18:38.547 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:18:38.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.667+0000 7ff5cdd8d700 1 Processor -- start 2023-11-08T08:18:38.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.668+0000 7ff5cdd8d700 1 -- start start 2023-11-08T08:18:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.668+0000 7ff5cdd8d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 0x7ff5c806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.668+0000 7ff5cdd8d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff5c80d15f0 0x7ff5c815b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.668+0000 7ff5cdd8d700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c815af30 con 0x7ff5c815e010 2023-11-08T08:18:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.668+0000 7ff5cdd8d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c815a9d0 con 0x7ff5c80d15f0 2023-11-08T08:18:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.668+0000 7ff5cdd8d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81584e0 con 0x7ff5c806e400 2023-11-08T08:18:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c7fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff5c815e010 0x7ff5c80ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58830/0 (socket says 172.21.15.116:58830) 2023-11-08T08:18:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c7fff700 1 -- 172.21.15.116:0/1143066962 learned_addr learned my addr 172.21.15.116:0/1143066962 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5beffd700 1 --2- 172.21.15.116:0/1143066962 >> v2:172.21.15.129:3300/0 conn(0x7ff5c80d15f0 0x7ff5c815b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824697707 0 0) 0x7ff5c815af30 con 0x7ff5c815e010 2023-11-08T08:18:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 0x7ff5c806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b40035b0 con 0x7ff5c815e010 2023-11-08T08:18:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225365768 0 0) 0x7ff5b40035b0 con 0x7ff5c815e010 2023-11-08T08:18:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 msgr2=0x7ff5c806e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 0x7ff5c806e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 >> v2:172.21.15.129:3300/0 conn(0x7ff5c80d15f0 msgr2=0x7ff5c815b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 --2- 172.21.15.116:0/1143066962 >> v2:172.21.15.129:3300/0 conn(0x7ff5c80d15f0 0x7ff5c815b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c815ef40 con 0x7ff5c815e010 2023-11-08T08:18:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7ff5c00019f0 con 0x7ff5c815e010 2023-11-08T08:18:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 >> v1:172.21.15.116:6789/0 conn(0x7ff5c815e010 legacy=0x7ff5c80ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5b4003a00 0x7ff5b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 0x7ff5b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4004d90 0x7ff5b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81584e0 con 0x7ff5b4003a00 2023-11-08T08:18:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b400ba10 con 0x7ff5b4004d90 2023-11-08T08:18:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b400bb50 con 0x7ff5c806e400 2023-11-08T08:18:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5b4003a00 0x7ff5b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5beffd700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 0x7ff5b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c7fff700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4004d90 0x7ff5b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 -- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 msgr2=0x7ff5b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c806e400 0x7ff5b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 -- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4004d90 msgr2=0x7ff5b4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4004d90 0x7ff5b4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c77fe700 1 -- 172.21.15.116:0/1143066962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40035b0 con 0x7ff5b4003a00 2023-11-08T08:18:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.669+0000 7ff5c7fff700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4004d90 0x7ff5b4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5b4003a00 0x7ff5b4003e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009b60 tx=0x7ff5b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3c3b3c59ba7d46 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800e660 con 0x7ff5b4003a00 2023-11-08T08:18:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800e7c0 con 0x7ff5b4003a00 2023-11-08T08:18:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5c6ffd700 1 -- 172.21.15.116:0/1143066962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800eb30 con 0x7ff5b4003a00 2023-11-08T08:18:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/1143066962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5b4003a00 msgr2=0x7ff5b4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/1143066962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5b4003a00 0x7ff5b4003e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009b60 tx=0x7ff5b4005880 comp rx=0 tx=0).stop 2023-11-08T08:18:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5c7fff700 1 -- 172.21.15.116:0/1143066962 reap_dead start 2023-11-08T08:18:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/1143066962 shutdown_connections 2023-11-08T08:18:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/1143066962 >> 172.21.15.116:0/1143066962 conn(0x7ff5c806a270 msgr2=0x7ff5c806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/1143066962 shutdown_connections 2023-11-08T08:18:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.670+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/1143066962 wait complete. 2023-11-08T08:18:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.671+0000 7ff5cdd8d700 1 Processor -- start 2023-11-08T08:18:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 -- start start 2023-11-08T08:18:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4003a00 0x7ff5c81f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c80b2f60 0x7ff5c8159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 0x7ff5c8144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c8144d90 con 0x7ff5c81443e0 2023-11-08T08:18:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80b3720 con 0x7ff5b4003a00 2023-11-08T08:18:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5cdd8d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80b0800 con 0x7ff5c80b2f60 2023-11-08T08:18:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5c7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 0x7ff5c8144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5c77fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4003a00 0x7ff5c81f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5c7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 0x7ff5c8144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52166/0 (socket says 172.21.15.116:52166) 2023-11-08T08:18:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5beffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c80b2f60 0x7ff5c8159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5c7fff700 1 -- 172.21.15.116:0/3342460766 learned_addr learned my addr 172.21.15.116:0/3342460766 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.672+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4003a00 0x7ff5c81f48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:39408/0 (socket says 172.21.15.116:39408) 2023-11-08T08:18:38.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c7fff700 1 -- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c80b2f60 msgr2=0x7ff5c8159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c7fff700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c80b2f60 0x7ff5c8159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c7fff700 1 -- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4003a00 msgr2=0x7ff5c81f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c7fff700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4003a00 0x7ff5c81f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c7fff700 1 -- 172.21.15.116:0/3342460766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c815ec60 con 0x7ff5c81443e0 2023-11-08T08:18:38.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c7fff700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 0x7ff5c8144850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0000f80 tx=0x7ff5c000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000ec10 con 0x7ff5c81443e0 2023-11-08T08:18:38.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c000ed70 con 0x7ff5c81443e0 2023-11-08T08:18:38.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c815e980 con 0x7ff5c81443e0 2023-11-08T08:18:38.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000eee0 con 0x7ff5c81443e0 2023-11-08T08:18:38.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.673+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c815abc0 con 0x7ff5c81443e0 2023-11-08T08:18:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.674+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c80743f0 con 0x7ff5c81443e0 2023-11-08T08:18:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.674+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7ff5c0013660 con 0x7ff5c81443e0 2023-11-08T08:18:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.675+0000 7ff5c4ff9700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682a0 0x7ff5a806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.675+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff5c0085f20 con 0x7ff5c81443e0 2023-11-08T08:18:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.675+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682a0 0x7ff5a806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.675+0000 7ff5c77fe700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682a0 0x7ff5a806a750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80090b0 tx=0x7ff5b801f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.679+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5c0052b20 con 0x7ff5c81443e0 2023-11-08T08:18:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.901+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5c806dcd0 con 0x7ff5c81443e0 2023-11-08T08:18:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.901+0000 7ff5c4ff9700 1 -- 172.21.15.116:0/3342460766 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7ff5c0056140 con 0x7ff5c81443e0 2023-11-08T08:18:38.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:38.903 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-11-08T08:18:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682a0 msgr2=0x7ff5a806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682a0 0x7ff5a806a750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80090b0 tx=0x7ff5b801f040 comp rx=0 tx=0).stop 2023-11-08T08:18:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 msgr2=0x7ff5c8144850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 0x7ff5c8144850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0000f80 tx=0x7ff5c000e440 comp rx=0 tx=0).stop 2023-11-08T08:18:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 shutdown_connections 2023-11-08T08:18:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c81443e0 0x7ff5c8144850 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682a0 0x7ff5a806a750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5b4003a00 0x7ff5c81f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 --2- 172.21.15.116:0/3342460766 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c80b2f60 0x7ff5c8159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 >> 172.21.15.116:0/3342460766 conn(0x7ff5c806a270 msgr2=0x7ff5c8154f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 shutdown_connections 2023-11-08T08:18:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:38.903+0000 7ff5cdd8d700 1 -- 172.21.15.116:0/3342460766 wait complete. 2023-11-08T08:18:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2023-11-08T08:18:38.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.039+0000 7efdbcce7700 1 Processor -- start 2023-11-08T08:18:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.039+0000 7efdbcce7700 1 -- start start 2023-11-08T08:18:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdbcce7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb80ce040 0x7efdb815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdbcce7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdb80b1770 0x7efdb80b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdbcce7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb813fc90 con 0x7efdb8142db0 2023-11-08T08:18:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdbcce7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80b2130 con 0x7efdb80b1770 2023-11-08T08:18:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdbcce7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb813f480 con 0x7efdb80ce040 2023-11-08T08:18:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdb80b1770 0x7efdb80b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb5d9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efdb8142db0 0x7efdb813ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58832/0 (socket says 172.21.15.116:58832) 2023-11-08T08:18:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdb80b1770 0x7efdb80b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52178/0 (socket says 172.21.15.116:52178) 2023-11-08T08:18:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb5d9b700 1 -- 172.21.15.116:0/883148689 learned_addr learned my addr 172.21.15.116:0/883148689 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb659c700 1 --2- 172.21.15.116:0/883148689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb80ce040 0x7efdb815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 -- 172.21.15.116:0/883148689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb80ce040 msgr2=0x7efdb815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 --2- 172.21.15.116:0/883148689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb80ce040 0x7efdb815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 -- 172.21.15.116:0/883148689 >> v1:172.21.15.116:6789/0 conn(0x7efdb8142db0 legacy=0x7efdb813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 -- 172.21.15.116:0/883148689 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb813f6d0 con 0x7efdb80b1770 2023-11-08T08:18:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb559a700 1 -- 172.21.15.116:0/883148689 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740512160 0 0) 0x7efdb813fc90 con 0x7efdb8142db0 2023-11-08T08:18:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb6d9d700 1 --2- 172.21.15.116:0/883148689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb80b1bf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009d90 tx=0x7efdb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b039343c36468f4b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.040+0000 7efdb559a700 1 -- 172.21.15.116:0/883148689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000e070 con 0x7efdb80b1770 2023-11-08T08:18:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdb559a700 1 -- 172.21.15.116:0/883148689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb000bb20 con 0x7efdb80b1770 2023-11-08T08:18:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdb559a700 1 -- 172.21.15.116:0/883148689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0012430 con 0x7efdb80b1770 2023-11-08T08:18:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 -- 172.21.15.116:0/883148689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 msgr2=0x7efdb80b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 --2- 172.21.15.116:0/883148689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb80b1bf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009d90 tx=0x7efdb000b330 comp rx=0 tx=0).stop 2023-11-08T08:18:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 -- 172.21.15.116:0/883148689 shutdown_connections 2023-11-08T08:18:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 --2- 172.21.15.116:0/883148689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb80b1bf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 --2- 172.21.15.116:0/883148689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb80ce040 0x7efdb815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 -- 172.21.15.116:0/883148689 >> 172.21.15.116:0/883148689 conn(0x7efdb806c010 msgr2=0x7efdb80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 -- 172.21.15.116:0/883148689 shutdown_connections 2023-11-08T08:18:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.041+0000 7efdbcce7700 1 -- 172.21.15.116:0/883148689 wait complete. 2023-11-08T08:18:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.043+0000 7efdbcce7700 1 Processor -- start 2023-11-08T08:18:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.043+0000 7efdbcce7700 1 -- start start 2023-11-08T08:18:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb8152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb80ce040 0x7efdb815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb8142db0 0x7efdb806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb806a3a0 con 0x7efdb80b1770 2023-11-08T08:18:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81e3c30 con 0x7efdb80ce040 2023-11-08T08:18:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81e3f10 con 0x7efdb8142db0 2023-11-08T08:18:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb8152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb8152a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52188/0 (socket says 172.21.15.116:52188) 2023-11-08T08:18:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 -- 172.21.15.116:0/3152084807 learned_addr learned my addr 172.21.15.116:0/3152084807 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb6d9d700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb8142db0 0x7efdb806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb5d9b700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb80ce040 0x7efdb815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 -- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb8142db0 msgr2=0x7efdb806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb8142db0 0x7efdb806e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 -- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb80ce040 msgr2=0x7efdb815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb80ce040 0x7efdb815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 -- 172.21.15.116:0/3152084807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0009a70 con 0x7efdb80b1770 2023-11-08T08:18:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb5d9b700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb80ce040 0x7efdb815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdb659c700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb8152a90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efda800e9b0 tx=0x7efda800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8009710 con 0x7efdb80b1770 2023-11-08T08:18:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda800c850 con 0x7efdb80b1770 2023-11-08T08:18:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb815d100 con 0x7efdb80b1770 2023-11-08T08:18:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.044+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda800cbc0 con 0x7efdb80b1770 2023-11-08T08:18:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.045+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb813f9b0 con 0x7efdb80b1770 2023-11-08T08:18:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.045+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb815d3e0 con 0x7efdb80b1770 2023-11-08T08:18:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.046+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efda8019030 con 0x7efdb80b1770 2023-11-08T08:18:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.046+0000 7efda6ffd700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd980682f0 0x7efd9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.046+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efda80858c0 con 0x7efdb80b1770 2023-11-08T08:18:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.046+0000 7efdb5d9b700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd980682f0 0x7efd9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.047+0000 7efdb5d9b700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd980682f0 0x7efd9806a7a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efdb81406e0 tx=0x7efdac007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.051+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efda8052470 con 0x7efdb80b1770 2023-11-08T08:18:39.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.231+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdb806a610 con 0x7efdb80b1770 2023-11-08T08:18:39.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.232+0000 7efda6ffd700 1 -- 172.21.15.116:0/3152084807 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+14255 (secure 0 0 0) 0x7efda8017090 con 0x7efdb80b1770 2023-11-08T08:18:39.234 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:39.234 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:38.458728+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:39.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd980682f0 msgr2=0x7efd9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd980682f0 0x7efd9806a7a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efdb81406e0 tx=0x7efdac007ca0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 msgr2=0x7efdb8152a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb8152a90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efda800e9b0 tx=0x7efda800ecc0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 shutdown_connections 2023-11-08T08:18:39.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb8142db0 0x7efdb806e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb80b1770 0x7efdb8152a90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd980682f0 0x7efd9806a7a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 --2- 172.21.15.116:0/3152084807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb80ce040 0x7efdb815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 >> 172.21.15.116:0/3152084807 conn(0x7efdb806c010 msgr2=0x7efdb8070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:39.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 shutdown_connections 2023-11-08T08:18:39.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.234+0000 7efdbcce7700 1 -- 172.21.15.116:0/3152084807 wait complete. 2023-11-08T08:18:39.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:18:39.248 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:18:39.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.369+0000 7f5bf243e700 1 Processor -- start 2023-11-08T08:18:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.369+0000 7f5bf243e700 1 -- start start 2023-11-08T08:18:39.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bf243e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5bec071890 0x7f5bec0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bf243e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bec15cc10 con 0x7f5bec0cf710 2023-11-08T08:18:39.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bf243e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bec06c9b0 con 0x7f5bec06a980 2023-11-08T08:18:39.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bf243e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bec06cc90 con 0x7f5bec071890 2023-11-08T08:18:39.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bf09db700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5bec071890 0x7f5bec0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bebfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5bec0cf710 0x7f5bec06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58842/0 (socket says 172.21.15.116:58842) 2023-11-08T08:18:39.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.370+0000 7f5bebfff700 1 -- 172.21.15.116:0/3337021484 learned_addr learned my addr 172.21.15.116:0/3337021484 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:39.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf09db700 1 -- 172.21.15.116:0/3337021484 >> v1:172.21.15.129:6789/0 conn(0x7f5bec06a980 legacy=0x7f5bec160480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:18:39.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf09db700 1 -- 172.21.15.116:0/3337021484 >> v1:172.21.15.116:6789/0 conn(0x7f5bec0cf710 legacy=0x7f5bec06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf09db700 1 -- 172.21.15.116:0/3337021484 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec06cf70 con 0x7f5bec071890 2023-11-08T08:18:39.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf09db700 1 --2- 172.21.15.116:0/3337021484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec0ce740 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009d90 tx=0x7f5be400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ea66827039d93d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5beaffd700 1 -- 172.21.15.116:0/3337021484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400e070 con 0x7f5bec071890 2023-11-08T08:18:39.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5beaffd700 1 -- 172.21.15.116:0/3337021484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be400bb90 con 0x7f5bec071890 2023-11-08T08:18:39.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5beaffd700 1 -- 172.21.15.116:0/3337021484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4003a20 con 0x7f5bec071890 2023-11-08T08:18:39.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 -- 172.21.15.116:0/3337021484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 msgr2=0x7f5bec0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 --2- 172.21.15.116:0/3337021484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec0ce740 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009d90 tx=0x7f5be400b330 comp rx=0 tx=0).stop 2023-11-08T08:18:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 -- 172.21.15.116:0/3337021484 shutdown_connections 2023-11-08T08:18:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 --2- 172.21.15.116:0/3337021484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec0ce740 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 -- 172.21.15.116:0/3337021484 >> 172.21.15.116:0/3337021484 conn(0x7f5bec06c010 msgr2=0x7f5bec0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:39.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 -- 172.21.15.116:0/3337021484 shutdown_connections 2023-11-08T08:18:39.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.371+0000 7f5bf243e700 1 -- 172.21.15.116:0/3337021484 wait complete. 2023-11-08T08:18:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 Processor -- start 2023-11-08T08:18:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 -- start start 2023-11-08T08:18:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bec06a980 0x7f5bec155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bec0cf710 0x7f5bec15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec15fc50 con 0x7f5bec071890 2023-11-08T08:18:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec15e990 con 0x7f5bec0cf710 2023-11-08T08:18:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.373+0000 7f5bf243e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec15f970 con 0x7f5bec06a980 2023-11-08T08:18:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52212/0 (socket says 172.21.15.116:52212) 2023-11-08T08:18:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 -- 172.21.15.116:0/1073490058 learned_addr learned my addr 172.21.15.116:0/1073490058 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5bf09db700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bec0cf710 0x7f5bec15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 -- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bec06a980 msgr2=0x7f5bec155c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bec06a980 0x7f5bec155c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 -- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bec0cf710 msgr2=0x7f5bec15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bec0cf710 0x7f5bec15e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 -- 172.21.15.116:0/1073490058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4009a70 con 0x7f5bec071890 2023-11-08T08:18:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5beb7fe700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec155680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc00b550 tx=0x7f5bdc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc011830 con 0x7f5bec071890 2023-11-08T08:18:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc0119b0 con 0x7f5bec071890 2023-11-08T08:18:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bec06d530 con 0x7f5bec071890 2023-11-08T08:18:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc011d60 con 0x7f5bec071890 2023-11-08T08:18:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.374+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bec06d250 con 0x7f5bec071890 2023-11-08T08:18:39.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.375+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bec15d7d0 con 0x7f5bec071890 2023-11-08T08:18:39.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.375+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5bdc021070 con 0x7f5bec071890 2023-11-08T08:18:39.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.376+0000 7f5be8ff9700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5bd80682f0 0x7f5bd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.376+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5bdc085860 con 0x7f5bec071890 2023-11-08T08:18:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.376+0000 7f5bebfff700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5bd80682f0 0x7f5bd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.376+0000 7f5bebfff700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5bd80682f0 0x7f5bd806a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5be0000f60 tx=0x7f5be0007a90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.380+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5bec15d7d0 con 0x7f5bec071890 2023-11-08T08:18:39.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.563+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5bec15d7d0 con 0x7f5bec071890 2023-11-08T08:18:39.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.563+0000 7f5be8ff9700 1 -- 172.21.15.116:0/1073490058 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5bec15d7d0 con 0x7f5bec071890 2023-11-08T08:18:39.565 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:18:39.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5bd80682f0 msgr2=0x7f5bd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5bd80682f0 0x7f5bd806a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5be0000f60 tx=0x7f5be0007a90 comp rx=0 tx=0).stop 2023-11-08T08:18:39.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 msgr2=0x7f5bec155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec155680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc00b550 tx=0x7f5bdc00b910 comp rx=0 tx=0).stop 2023-11-08T08:18:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 shutdown_connections 2023-11-08T08:18:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5bd80682f0 0x7f5bd806a7a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bec0cf710 0x7f5bec15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bec06a980 0x7f5bec155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 --2- 172.21.15.116:0/1073490058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec071890 0x7f5bec155680 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 >> 172.21.15.116:0/1073490058 conn(0x7f5bec06c010 msgr2=0x7f5bec0b4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:39.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 shutdown_connections 2023-11-08T08:18:39.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.565+0000 7f5bf243e700 1 -- 172.21.15.116:0/1073490058 wait complete. 2023-11-08T08:18:39.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:18:39.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 Processor -- start 2023-11-08T08:18:39.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 -- start start 2023-11-08T08:18:39.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f881015c810 0x7f8810160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f881006c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f881006cdc0 0x7f881006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881015cbe0 con 0x7f881015c810 2023-11-08T08:18:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88101607a0 con 0x7f881006cdc0 2023-11-08T08:18:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f8816b63700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8810160a50 con 0x7f88100718e0 2023-11-08T08:18:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.700+0000 7f88148ff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f881015c810 0x7f8810160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f881015c810 0x7f8810160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52214/0 (socket says 172.21.15.116:52214) 2023-11-08T08:18:39.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 -- 172.21.15.116:0/252636148 learned_addr learned my addr 172.21.15.116:0/252636148 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:39.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 -- 172.21.15.116:0/252636148 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 msgr2=0x7f881006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:39.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 --2- 172.21.15.116:0/252636148 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f881006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 -- 172.21.15.116:0/252636148 >> v2:172.21.15.129:3300/0 conn(0x7f881006cdc0 msgr2=0x7f881006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 --2- 172.21.15.116:0/252636148 >> v2:172.21.15.129:3300/0 conn(0x7f881006cdc0 0x7f881006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 -- 172.21.15.116:0/252636148 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8810160d30 con 0x7f881015c810 2023-11-08T08:18:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f88148ff700 1 --2- 172.21.15.116:0/252636148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f8810160260 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f88101612f0 tx=0x7f8804009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ab9c81acb4455e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f880f7fe700 1 -- 172.21.15.116:0/252636148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400b870 con 0x7f881015c810 2023-11-08T08:18:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f880f7fe700 1 -- 172.21.15.116:0/252636148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880400b9d0 con 0x7f881015c810 2023-11-08T08:18:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.701+0000 7f880f7fe700 1 -- 172.21.15.116:0/252636148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400bd60 con 0x7f881015c810 2023-11-08T08:18:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 -- 172.21.15.116:0/252636148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 msgr2=0x7f8810160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 --2- 172.21.15.116:0/252636148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f8810160260 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f88101612f0 tx=0x7f8804009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 -- 172.21.15.116:0/252636148 shutdown_connections 2023-11-08T08:18:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 --2- 172.21.15.116:0/252636148 >> v2:172.21.15.129:3300/0 conn(0x7f881006cdc0 0x7f881006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 --2- 172.21.15.116:0/252636148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f8810160260 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 --2- 172.21.15.116:0/252636148 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f881006c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 -- 172.21.15.116:0/252636148 >> 172.21.15.116:0/252636148 conn(0x7f881006c010 msgr2=0x7f88100cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 -- 172.21.15.116:0/252636148 shutdown_connections 2023-11-08T08:18:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.702+0000 7f8816b63700 1 -- 172.21.15.116:0/252636148 wait complete. 2023-11-08T08:18:39.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.703+0000 7f8816b63700 1 Processor -- start 2023-11-08T08:18:39.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.703+0000 7f8816b63700 1 -- start start 2023-11-08T08:18:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f881006cdc0 0x7f881014abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f88100c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f88100aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100c9640 con 0x7f881015c810 2023-11-08T08:18:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100c8060 con 0x7f881006cdc0 2023-11-08T08:18:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88100c8340 con 0x7f88100718e0 2023-11-08T08:18:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f88100aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f88100aecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52216/0 (socket says 172.21.15.116:52216) 2023-11-08T08:18:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 -- 172.21.15.116:0/2640600639 learned_addr learned my addr 172.21.15.116:0/2640600639 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f88148ff700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f881006cdc0 0x7f881014abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f880ffff700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f88100c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 -- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 msgr2=0x7f88100c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f88100c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 -- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f881006cdc0 msgr2=0x7f881014abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f881006cdc0 0x7f881014abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 -- 172.21.15.116:0/2640600639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8804009550 con 0x7f881015c810 2023-11-08T08:18:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8815100700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f88100aecd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f880800b9e0 tx=0x7f880800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800d8e0 con 0x7f881015c810 2023-11-08T08:18:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880800da40 con 0x7f881015c810 2023-11-08T08:18:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88100ceb20 con 0x7f881015c810 2023-11-08T08:18:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800ddf0 con 0x7f881015c810 2023-11-08T08:18:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.704+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8810161010 con 0x7f881015c810 2023-11-08T08:18:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.705+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100ce740 con 0x7f881015c810 2023-11-08T08:18:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.706+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8808015420 con 0x7f881015c810 2023-11-08T08:18:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.706+0000 7f880d7fa700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f87f40682f0 0x7f87f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.706+0000 7f88148ff700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f87f40682f0 0x7f87f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.706+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8808011070 con 0x7f881015c810 2023-11-08T08:18:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.706+0000 7f88148ff700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f87f40682f0 0x7f87f406a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8804014040 tx=0x7f8804008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.710+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f880804d5e0 con 0x7f881015c810 2023-11-08T08:18:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.890+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88100743f0 con 0x7f881015c810 2023-11-08T08:18:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.890+0000 7f880d7fa700 1 -- 172.21.15.116:0/2640600639 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8808052520 con 0x7f881015c810 2023-11-08T08:18:39.892 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:18:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f87f40682f0 msgr2=0x7f87f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f87f40682f0 0x7f87f406a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8804014040 tx=0x7f8804008030 comp rx=0 tx=0).stop 2023-11-08T08:18:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 msgr2=0x7f88100aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f88100aecd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f880800b9e0 tx=0x7f880800bcf0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 shutdown_connections 2023-11-08T08:18:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881015c810 0x7f88100aecd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f87f40682f0 0x7f87f406a7a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f881006cdc0 0x7f881014abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 --2- 172.21.15.116:0/2640600639 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f88100718e0 0x7f88100c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 >> 172.21.15.116:0/2640600639 conn(0x7f881006c010 msgr2=0x7f88100cc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 shutdown_connections 2023-11-08T08:18:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:39.893+0000 7f8816b63700 1 -- 172.21.15.116:0/2640600639 wait complete. 2023-11-08T08:18:39.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-08T08:18:40.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 Processor -- start 2023-11-08T08:18:40.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 -- start start 2023-11-08T08:18:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f28180b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28180ce1e0 0x7f28180b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28180b4fc0 con 0x7f281806feb0 2023-11-08T08:18:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2818159c30 con 0x7f28180ce1e0 2023-11-08T08:18:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.024+0000 7f281e4b0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28180b05e0 con 0x7f28180ced70 2023-11-08T08:18:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28180ce1e0 0x7f28180b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28180ce1e0 0x7f28180b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52222/0 (socket says 172.21.15.116:52222) 2023-11-08T08:18:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 -- 172.21.15.116:0/2511956486 learned_addr learned my addr 172.21.15.116:0/2511956486 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f2817fff700 1 --2- 172.21.15.116:0/2511956486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f28180b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 -- 172.21.15.116:0/2511956486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 msgr2=0x7f28180b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 --2- 172.21.15.116:0/2511956486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f28180b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 -- 172.21.15.116:0/2511956486 >> v1:172.21.15.129:6789/0 conn(0x7f281806feb0 legacy=0x7f281806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 -- 172.21.15.116:0/2511956486 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180b08c0 con 0x7f28180ce1e0 2023-11-08T08:18:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f281ca4d700 1 --2- 172.21.15.116:0/2511956486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28180ce1e0 0x7f28180b4930 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f281000d150 tx=0x7f281000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8195ec1b2d67ab4 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f2816ffd700 1 -- 172.21.15.116:0/2511956486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810010070 con 0x7f28180ce1e0 2023-11-08T08:18:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f2816ffd700 1 -- 172.21.15.116:0/2511956486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281000dd80 con 0x7f28180ce1e0 2023-11-08T08:18:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.025+0000 7f2816ffd700 1 -- 172.21.15.116:0/2511956486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281000b310 con 0x7f28180ce1e0 2023-11-08T08:18:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 -- 172.21.15.116:0/2511956486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28180ce1e0 msgr2=0x7f28180b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 --2- 172.21.15.116:0/2511956486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28180ce1e0 0x7f28180b4930 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f281000d150 tx=0x7f281000d460 comp rx=0 tx=0).stop 2023-11-08T08:18:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 -- 172.21.15.116:0/2511956486 shutdown_connections 2023-11-08T08:18:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 --2- 172.21.15.116:0/2511956486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28180ce1e0 0x7f28180b4930 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 --2- 172.21.15.116:0/2511956486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f28180b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 -- 172.21.15.116:0/2511956486 >> 172.21.15.116:0/2511956486 conn(0x7f28180b2fc0 msgr2=0x7f2818071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 -- 172.21.15.116:0/2511956486 shutdown_connections 2023-11-08T08:18:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.026+0000 7f281e4b0700 1 -- 172.21.15.116:0/2511956486 wait complete. 2023-11-08T08:18:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.027+0000 7f281e4b0700 1 Processor -- start 2023-11-08T08:18:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.027+0000 7f281e4b0700 1 -- start start 2023-11-08T08:18:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 0x7f28181f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28180ce1e0 0x7f2818159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f281815a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28181f2220 con 0x7f281806feb0 2023-11-08T08:18:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281815c820 con 0x7f28180ce1e0 2023-11-08T08:18:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281815cb00 con 0x7f28180ced70 2023-11-08T08:18:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 0x7f28181f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 0x7f28181f1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52224/0 (socket says 172.21.15.116:52224) 2023-11-08T08:18:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 -- 172.21.15.116:0/4126416006 learned_addr learned my addr 172.21.15.116:0/4126416006 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281ca4d700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f281815a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f28177fe700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28180ce1e0 0x7f2818159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 -- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 msgr2=0x7f281815a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f281815a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 -- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28180ce1e0 msgr2=0x7f2818159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28180ce1e0 0x7f2818159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 -- 172.21.15.116:0/4126416006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281000bd40 con 0x7f281806feb0 2023-11-08T08:18:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281ca4d700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f281815a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f28177fe700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28180ce1e0 0x7f2818159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2817fff700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 0x7f28181f1ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f280800c7f0 tx=0x7f280800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808011800 con 0x7f281806feb0 2023-11-08T08:18:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808011960 con 0x7f281806feb0 2023-11-08T08:18:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28180b5260 con 0x7f281806feb0 2023-11-08T08:18:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808011d10 con 0x7f281806feb0 2023-11-08T08:18:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.028+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28180b0ba0 con 0x7f281806feb0 2023-11-08T08:18:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.029+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2818072420 con 0x7f281806feb0 2023-11-08T08:18:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.030+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2808021070 con 0x7f281806feb0 2023-11-08T08:18:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.030+0000 7f2814ff9700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f28040682f0 0x7f280406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.030+0000 7f28177fe700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f28040682f0 0x7f280406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.030+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2808085770 con 0x7f281806feb0 2023-11-08T08:18:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.030+0000 7f28177fe700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f28040682f0 0x7f280406a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2818158980 tx=0x7f280c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.034+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2808052320 con 0x7f281806feb0 2023-11-08T08:18:40.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.216+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2818067370 con 0x7f281806feb0 2023-11-08T08:18:40.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.216+0000 7f2814ff9700 1 -- 172.21.15.116:0/4126416006 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+126 (secure 0 0 0) 0x7f280801f090 con 0x7f281806feb0 2023-11-08T08:18:40.218 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-08T08:18:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:18:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-08T08:18:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": "rbd" 2023-11-08T08:18:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:18:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 2, 2023-11-08T08:18:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-08T08:18:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:18:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-08T08:18:40.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f28040682f0 msgr2=0x7f280406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f28040682f0 0x7f280406a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2818158980 tx=0x7f280c009040 comp rx=0 tx=0).stop 2023-11-08T08:18:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 msgr2=0x7f28181f1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 0x7f28181f1ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f280800c7f0 tx=0x7f280800cbb0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 shutdown_connections 2023-11-08T08:18:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f28180ced70 0x7f281815a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.218+0000 7f281e4b0700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f281806feb0 0x7f28181f1ce0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.219+0000 7f281e4b0700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f28040682f0 0x7f280406a7a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.219+0000 7f281e4b0700 1 --2- 172.21.15.116:0/4126416006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28180ce1e0 0x7f2818159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.219+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 >> 172.21.15.116:0/4126416006 conn(0x7f28180b2fc0 msgr2=0x7f281806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.219+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 shutdown_connections 2023-11-08T08:18:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.219+0000 7f281e4b0700 1 -- 172.21.15.116:0/4126416006 wait complete. 2023-11-08T08:18:40.231 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-08T08:18:40.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-08T08:18:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.354+0000 7fdb42aef700 1 Processor -- start 2023-11-08T08:18:40.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.354+0000 7fdb42aef700 1 -- start start 2023-11-08T08:18:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb42aef700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb3c0ae7a0 0x7fdb3c160920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb42aef700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdb3c160e60 0x7fdb3c0ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb42aef700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0cb420 con 0x7fdb3c15c620 2023-11-08T08:18:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb42aef700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c066620 con 0x7fdb3c0ae7a0 2023-11-08T08:18:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb42aef700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c0cb700 con 0x7fdb3c160e60 2023-11-08T08:18:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb3bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb3c0ae7a0 0x7fdb3c160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb4088b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdb3c15c620 0x7fdb3c0cac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58856/0 (socket says 172.21.15.116:58856) 2023-11-08T08:18:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb4088b700 1 -- 172.21.15.116:0/4277699796 learned_addr learned my addr 172.21.15.116:0/4277699796 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb4108c700 1 --2- 172.21.15.116:0/4277699796 >> v2:172.21.15.129:3300/0 conn(0x7fdb3c160e60 0x7fdb3c0ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb3bfff700 1 -- 172.21.15.116:0/4277699796 >> v2:172.21.15.129:3300/0 conn(0x7fdb3c160e60 msgr2=0x7fdb3c0ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:40.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb3bfff700 1 --2- 172.21.15.116:0/4277699796 >> v2:172.21.15.129:3300/0 conn(0x7fdb3c160e60 0x7fdb3c0ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb3bfff700 1 -- 172.21.15.116:0/4277699796 >> v1:172.21.15.116:6789/0 conn(0x7fdb3c15c620 legacy=0x7fdb3c0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb3bfff700 1 -- 172.21.15.116:0/4277699796 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0cb9e0 con 0x7fdb3c0ae7a0 2023-11-08T08:18:40.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb4108c700 1 --2- 172.21.15.116:0/4277699796 >> v2:172.21.15.129:3300/0 conn(0x7fdb3c160e60 0x7fdb3c0ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.355+0000 7fdb3b7fe700 1 -- 172.21.15.116:0/4277699796 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509857152 0 0) 0x7fdb3c0cb420 con 0x7fdb3c15c620 2023-11-08T08:18:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb3bfff700 1 --2- 172.21.15.116:0/4277699796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c0ae7a0 0x7fdb3c160920 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c06bd50 tx=0x7fdb24009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34646433844c0a6d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb3b7fe700 1 -- 172.21.15.116:0/4277699796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24011830 con 0x7fdb3c0ae7a0 2023-11-08T08:18:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb3b7fe700 1 -- 172.21.15.116:0/4277699796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb24011990 con 0x7fdb3c0ae7a0 2023-11-08T08:18:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb3b7fe700 1 -- 172.21.15.116:0/4277699796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24011d20 con 0x7fdb3c0ae7a0 2023-11-08T08:18:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 -- 172.21.15.116:0/4277699796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c0ae7a0 msgr2=0x7fdb3c160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 --2- 172.21.15.116:0/4277699796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c0ae7a0 0x7fdb3c160920 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c06bd50 tx=0x7fdb24009b20 comp rx=0 tx=0).stop 2023-11-08T08:18:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 -- 172.21.15.116:0/4277699796 shutdown_connections 2023-11-08T08:18:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 --2- 172.21.15.116:0/4277699796 >> v2:172.21.15.129:3300/0 conn(0x7fdb3c160e60 0x7fdb3c0ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 --2- 172.21.15.116:0/4277699796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c0ae7a0 0x7fdb3c160920 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 -- 172.21.15.116:0/4277699796 >> 172.21.15.116:0/4277699796 conn(0x7fdb3c0cf370 msgr2=0x7fdb3c0cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 -- 172.21.15.116:0/4277699796 shutdown_connections 2023-11-08T08:18:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.356+0000 7fdb42aef700 1 -- 172.21.15.116:0/4277699796 wait complete. 2023-11-08T08:18:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.357+0000 7fdb42aef700 1 Processor -- start 2023-11-08T08:18:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.357+0000 7fdb42aef700 1 -- start start 2023-11-08T08:18:40.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb42aef700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb3c0ae7a0 0x7fdb3c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb42aef700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 0x7fdb3c1487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb42aef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdb3c160e60 0x7fdb3c14b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb42aef700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c14b180 con 0x7fdb3c15c620 2023-11-08T08:18:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb42aef700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0b2a30 con 0x7fdb3c0ae7a0 2023-11-08T08:18:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb42aef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0b2d10 con 0x7fdb3c160e60 2023-11-08T08:18:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb3bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 0x7fdb3c1487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb3bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 0x7fdb3c1487a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52244/0 (socket says 172.21.15.116:52244) 2023-11-08T08:18:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.358+0000 7fdb3bfff700 1 -- 172.21.15.116:0/1165963454 learned_addr learned my addr 172.21.15.116:0/1165963454 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb3bfff700 1 -- 172.21.15.116:0/1165963454 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdb3c160e60 msgr2=0x7fdb3c14b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb3bfff700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdb3c160e60 0x7fdb3c14b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb3bfff700 1 -- 172.21.15.116:0/1165963454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb3c0ae7a0 msgr2=0x7fdb3c0b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb3bfff700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb3c0ae7a0 0x7fdb3c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb3bfff700 1 -- 172.21.15.116:0/1165963454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240094a0 con 0x7fdb3c15c620 2023-11-08T08:18:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb3bfff700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 0x7fdb3c1487a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb24000c00 tx=0x7fdb24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb240038a0 con 0x7fdb3c15c620 2023-11-08T08:18:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2401f070 con 0x7fdb3c15c620 2023-11-08T08:18:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24003c40 con 0x7fdb3c15c620 2023-11-08T08:18:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c15c9f0 con 0x7fdb3c15c620 2023-11-08T08:18:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.359+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c160400 con 0x7fdb3c15c620 2023-11-08T08:18:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.360+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c15cb80 con 0x7fdb3c15c620 2023-11-08T08:18:40.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.360+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb2400f050 con 0x7fdb3c15c620 2023-11-08T08:18:40.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.360+0000 7fdb397fa700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdb1c0682f0 0x7fdb1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:40.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.361+0000 7fdb4088b700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdb1c0682f0 0x7fdb1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:40.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.361+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdb24021070 con 0x7fdb3c15c620 2023-11-08T08:18:40.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.361+0000 7fdb4088b700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdb1c0682f0 0x7fdb1c06a7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb34001560 tx=0x7fdb34001740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:40.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.365+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb24053fc0 con 0x7fdb3c15c620 2023-11-08T08:18:40.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:40.549+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdb3c0743f0 con 0x7fdb3c15c620 2023-11-08T08:18:41.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.479+0000 7fdb397fa700 1 -- 172.21.15.116:0/1165963454 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb240150b0 con 0x7fdb3c15c620 2023-11-08T08:18:41.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.480+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdb1c0682f0 msgr2=0x7fdb1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:41.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.480+0000 7fdb42aef700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdb1c0682f0 0x7fdb1c06a7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb34001560 tx=0x7fdb34001740 comp rx=0 tx=0).stop 2023-11-08T08:18:41.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.480+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 msgr2=0x7fdb3c1487a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:41.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.480+0000 7fdb42aef700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 0x7fdb3c1487a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb24000c00 tx=0x7fdb24004750 comp rx=0 tx=0).stop 2023-11-08T08:18:41.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 shutdown_connections 2023-11-08T08:18:41.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdb3c160e60 0x7fdb3c14b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdb1c0682f0 0x7fdb1c06a7a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb3c0ae7a0 0x7fdb3c0b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 --2- 172.21.15.116:0/1165963454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb3c15c620 0x7fdb3c1487a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 >> 172.21.15.116:0/1165963454 conn(0x7fdb3c0cf370 msgr2=0x7fdb3c070310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:41.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 shutdown_connections 2023-11-08T08:18:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.481+0000 7fdb42aef700 1 -- 172.21.15.116:0/1165963454 wait complete. 2023-11-08T08:18:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-08T08:18:41.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-08T08:18:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.627+0000 7ff593845700 1 Processor -- start 2023-11-08T08:18:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.627+0000 7ff593845700 1 -- start start 2023-11-08T08:18:41.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff593845700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c0718e0 0x7ff58c06c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:41.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff593845700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff58c06cdc0 0x7ff58c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff593845700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c160a50 con 0x7ff58c15c810 2023-11-08T08:18:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff593845700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c15cbe0 con 0x7ff58c06cdc0 2023-11-08T08:18:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff593845700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c160d30 con 0x7ff58c0718e0 2023-11-08T08:18:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff591de2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff58c06cdc0 0x7ff58c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff5915e1700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff58c15c810 0x7ff58c160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58864/0 (socket says 172.21.15.116:58864) 2023-11-08T08:18:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff5915e1700 1 -- 172.21.15.116:0/1469022838 learned_addr learned my addr 172.21.15.116:0/1469022838 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff590de0700 1 --2- 172.21.15.116:0/1469022838 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c0718e0 0x7ff58c06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff591de2700 1 -- 172.21.15.116:0/1469022838 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c0718e0 msgr2=0x7ff58c06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff591de2700 1 --2- 172.21.15.116:0/1469022838 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c0718e0 0x7ff58c06c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff591de2700 1 -- 172.21.15.116:0/1469022838 >> v1:172.21.15.116:6789/0 conn(0x7ff58c15c810 legacy=0x7ff58c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff583fff700 1 -- 172.21.15.116:0/1469022838 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976338052 0 0) 0x7ff58c160a50 con 0x7ff58c15c810 2023-11-08T08:18:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff591de2700 1 -- 172.21.15.116:0/1469022838 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c161010 con 0x7ff58c06cdc0 2023-11-08T08:18:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff591de2700 1 --2- 172.21.15.116:0/1469022838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c06cdc0 0x7ff58c06a270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff584009d90 tx=0x7ff58400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971684bb3270b739 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff583fff700 1 -- 172.21.15.116:0/1469022838 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58400e070 con 0x7ff58c06cdc0 2023-11-08T08:18:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff583fff700 1 -- 172.21.15.116:0/1469022838 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58400bbd0 con 0x7ff58c06cdc0 2023-11-08T08:18:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.628+0000 7ff583fff700 1 -- 172.21.15.116:0/1469022838 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584012430 con 0x7ff58c06cdc0 2023-11-08T08:18:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 -- 172.21.15.116:0/1469022838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c06cdc0 msgr2=0x7ff58c06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 --2- 172.21.15.116:0/1469022838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c06cdc0 0x7ff58c06a270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff584009d90 tx=0x7ff58400b330 comp rx=0 tx=0).stop 2023-11-08T08:18:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 -- 172.21.15.116:0/1469022838 shutdown_connections 2023-11-08T08:18:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 --2- 172.21.15.116:0/1469022838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c06cdc0 0x7ff58c06a270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 --2- 172.21.15.116:0/1469022838 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c0718e0 0x7ff58c06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 -- 172.21.15.116:0/1469022838 >> 172.21.15.116:0/1469022838 conn(0x7ff58c06c010 msgr2=0x7ff58c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:41.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 -- 172.21.15.116:0/1469022838 shutdown_connections 2023-11-08T08:18:41.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.629+0000 7ff593845700 1 -- 172.21.15.116:0/1469022838 wait complete. 2023-11-08T08:18:41.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.630+0000 7ff593845700 1 Processor -- start 2023-11-08T08:18:41.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.630+0000 7ff593845700 1 -- start start 2023-11-08T08:18:41.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff593845700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff58c06cdc0 0x7ff58c1f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:41.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff593845700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 0x7ff58c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:41.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff593845700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c15c810 0x7ff58c0cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:41.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff593845700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c152b70 con 0x7ff58c0718e0 2023-11-08T08:18:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff593845700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c1f1ce0 con 0x7ff58c06cdc0 2023-11-08T08:18:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff593845700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0c9190 con 0x7ff58c15c810 2023-11-08T08:18:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 0x7ff58c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 0x7ff58c0c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52262/0 (socket says 172.21.15.116:52262) 2023-11-08T08:18:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 -- 172.21.15.116:0/2808138648 learned_addr learned my addr 172.21.15.116:0/2808138648 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff5915e1700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff58c06cdc0 0x7ff58c1f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff591de2700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c15c810 0x7ff58c0cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 -- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c15c810 msgr2=0x7ff58c0cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c15c810 0x7ff58c0cb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 -- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff58c06cdc0 msgr2=0x7ff58c1f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff58c06cdc0 0x7ff58c1f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:41.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 -- 172.21.15.116:0/2808138648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff584009a70 con 0x7ff58c0718e0 2023-11-08T08:18:41.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff5915e1700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff58c06cdc0 0x7ff58c1f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:41.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.631+0000 7ff590de0700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 0x7ff58c0c4010 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00e810 tx=0x7ff57c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:41.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.632+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c013610 con 0x7ff58c0718e0 2023-11-08T08:18:41.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.632+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c013770 con 0x7ff58c0718e0 2023-11-08T08:18:41.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.632+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c0ce740 con 0x7ff58c0718e0 2023-11-08T08:18:41.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.632+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c013b20 con 0x7ff58c0718e0 2023-11-08T08:18:41.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.632+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c0ceb20 con 0x7ff58c0718e0 2023-11-08T08:18:41.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.633+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c0ce8d0 con 0x7ff58c0718e0 2023-11-08T08:18:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.633+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff57c023070 con 0x7ff58c0718e0 2023-11-08T08:18:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.633+0000 7ff581ffb700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5700682f0 0x7ff57006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.633+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5580+0+0 (secure 0 0 0) 0x7ff57c0857c0 con 0x7ff58c0718e0 2023-11-08T08:18:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.633+0000 7ff5915e1700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5700682f0 0x7ff57006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.634+0000 7ff5915e1700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5700682f0 0x7ff57006a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff588007fd0 tx=0x7ff588009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:41.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.639+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff57c052180 con 0x7ff58c0718e0 2023-11-08T08:18:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:41.818+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff58c0743f0 con 0x7ff58c0718e0 2023-11-08T08:18:42.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.490+0000 7ff581ffb700 1 -- 172.21.15.116:0/2808138648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff57c0557a0 con 0x7ff58c0718e0 2023-11-08T08:18:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.506+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5700682f0 msgr2=0x7ff57006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.506+0000 7ff593845700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5700682f0 0x7ff57006a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff588007fd0 tx=0x7ff588009040 comp rx=0 tx=0).stop 2023-11-08T08:18:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 msgr2=0x7ff58c0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 0x7ff58c0c4010 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00e810 tx=0x7ff57c00ebd0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 shutdown_connections 2023-11-08T08:18:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff58c15c810 0x7ff58c0cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5700682f0 0x7ff57006a7a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff58c06cdc0 0x7ff58c1f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 --2- 172.21.15.116:0/2808138648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff58c0718e0 0x7ff58c0c4010 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 >> 172.21.15.116:0/2808138648 conn(0x7ff58c06c010 msgr2=0x7ff58c158070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 shutdown_connections 2023-11-08T08:18:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.507+0000 7ff593845700 1 -- 172.21.15.116:0/2808138648 wait complete. 2023-11-08T08:18:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-08T08:18:42.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-08T08:18:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.648+0000 7fed97a0c700 1 Processor -- start 2023-11-08T08:18:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.648+0000 7fed97a0c700 1 -- start start 2023-11-08T08:18:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed97a0c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 0x7fed90160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed97a0c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900d2d30 con 0x7fed900b5dc0 2023-11-08T08:18:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed97a0c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900d0b40 con 0x7fed9006b220 2023-11-08T08:18:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed97a0c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed901611f0 con 0x7fed9015c8c0 2023-11-08T08:18:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed94fa7700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fed900b5dc0 0x7fed9006ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34240/0 (socket says 172.21.15.116:34240) 2023-11-08T08:18:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed94fa7700 1 -- 172.21.15.116:0/2787437057 learned_addr learned my addr 172.21.15.116:0/2787437057 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed957a8700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 0x7fed90160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198218199 0 0) 0x7fed84001110 con 0x7fed900b5dc0 2023-11-08T08:18:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.649+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed780035b0 con 0x7fed900b5dc0 2023-11-08T08:18:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388798447 0 0) 0x7fed780035b0 con 0x7fed900b5dc0 2023-11-08T08:18:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 msgr2=0x7fed90160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 0x7fed90160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 >> v1:172.21.15.129:6789/0 conn(0x7fed9006b220 legacy=0x7fed900d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900d2a50 con 0x7fed900b5dc0 2023-11-08T08:18:42.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fed84001800 con 0x7fed900b5dc0 2023-11-08T08:18:42.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 >> v1:172.21.15.116:6789/0 conn(0x7fed900b5dc0 legacy=0x7fed9006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 0x7fed78003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003fb0 0x7fed78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004d90 0x7fed78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed901611f0 con 0x7fed78003fb0 2023-11-08T08:18:42.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7800baf0 con 0x7fed78004d90 2023-11-08T08:18:42.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7800bcb0 con 0x7fed9015c8c0 2023-11-08T08:18:42.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003fb0 0x7fed78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed957a8700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 0x7fed78003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed95fa9700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004d90 0x7fed78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 -- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 msgr2=0x7fed78003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:42.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed9015c8c0 0x7fed78003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 -- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004d90 msgr2=0x7fed78005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:42.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004d90 0x7fed78005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 -- 172.21.15.116:0/2787437057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780035b0 con 0x7fed78003fb0 2023-11-08T08:18:42.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed95fa9700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004d90 0x7fed78005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:42.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed94fa7700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003fb0 0x7fed78004850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fed8400c210 tx=0x7fed78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad80bb5307461e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:42.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84007db0 con 0x7fed78003fb0 2023-11-08T08:18:42.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8400cdd0 con 0x7fed78003fb0 2023-11-08T08:18:42.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.650+0000 7fed8ffff700 1 -- 172.21.15.116:0/2787437057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84005ba0 con 0x7fed78003fb0 2023-11-08T08:18:42.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed97a0c700 1 -- 172.21.15.116:0/2787437057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003fb0 msgr2=0x7fed78004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2787437057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003fb0 0x7fed78004850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fed8400c210 tx=0x7fed78005880 comp rx=0 tx=0).stop 2023-11-08T08:18:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed95fa9700 1 -- 172.21.15.116:0/2787437057 reap_dead start 2023-11-08T08:18:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed97a0c700 1 -- 172.21.15.116:0/2787437057 shutdown_connections 2023-11-08T08:18:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed97a0c700 1 -- 172.21.15.116:0/2787437057 >> 172.21.15.116:0/2787437057 conn(0x7fed9006c520 msgr2=0x7fed9006c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed97a0c700 1 -- 172.21.15.116:0/2787437057 shutdown_connections 2023-11-08T08:18:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.651+0000 7fed97a0c700 1 -- 172.21.15.116:0/2787437057 wait complete. 2023-11-08T08:18:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.652+0000 7fed97a0c700 1 Processor -- start 2023-11-08T08:18:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.652+0000 7fed97a0c700 1 -- start start 2023-11-08T08:18:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed78003fb0 0x7fed901563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed9015f9b0 0x7fed901e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 0x7fed901e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed900b4100 con 0x7fed901e65d0 2023-11-08T08:18:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9015e910 con 0x7fed9015f9b0 2023-11-08T08:18:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9015ebf0 con 0x7fed78003fb0 2023-11-08T08:18:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 0x7fed901e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 0x7fed901e6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38958/0 (socket says 172.21.15.116:38958) 2023-11-08T08:18:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 -- 172.21.15.116:0/2617310908 learned_addr learned my addr 172.21.15.116:0/2617310908 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed94fa7700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed9015f9b0 0x7fed901e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed957a8700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed78003fb0 0x7fed901563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 -- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed78003fb0 msgr2=0x7fed901563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed78003fb0 0x7fed901563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 -- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed9015f9b0 msgr2=0x7fed901e6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed9015f9b0 0x7fed901e6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 -- 172.21.15.116:0/2617310908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900cda40 con 0x7fed901e65d0 2023-11-08T08:18:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed957a8700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed78003fb0 0x7fed901563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed95fa9700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 0x7fed901e6ff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fed9006bfe0 tx=0x7fed8000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80003c10 con 0x7fed901e65d0 2023-11-08T08:18:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80003d90 con 0x7fed901e65d0 2023-11-08T08:18:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed900cd760 con 0x7fed901e65d0 2023-11-08T08:18:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.653+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80005310 con 0x7fed901e65d0 2023-11-08T08:18:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.654+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed900d2700 con 0x7fed901e65d0 2023-11-08T08:18:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.654+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed90067d70 con 0x7fed901e65d0 2023-11-08T08:18:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.655+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed80005470 con 0x7fed901e65d0 2023-11-08T08:18:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.655+0000 7fed8dffb700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed7c0682f0 0x7fed7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.655+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fed80085db0 con 0x7fed901e65d0 2023-11-08T08:18:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.655+0000 7fed957a8700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed7c0682f0 0x7fed7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.656+0000 7fed957a8700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed7c0682f0 0x7fed7c06a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed88009c60 tx=0x7fed88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.660+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed80052670 con 0x7fed901e65d0 2023-11-08T08:18:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.884+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fed900743f0 con 0x7fed901e65d0 2023-11-08T08:18:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.889+0000 7fed8dffb700 1 -- 172.21.15.116:0/2617310908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fed80001b50 con 0x7fed901e65d0 2023-11-08T08:18:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.895+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed7c0682f0 msgr2=0x7fed7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.895+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed7c0682f0 0x7fed7c06a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed88009c60 tx=0x7fed88008040 comp rx=0 tx=0).stop 2023-11-08T08:18:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.895+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 msgr2=0x7fed901e6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.895+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 0x7fed901e6ff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fed9006bfe0 tx=0x7fed8000edc0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.896+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 shutdown_connections 2023-11-08T08:18:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.896+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed901e65d0 0x7fed901e6ff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.896+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed78003fb0 0x7fed901563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.896+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed7c0682f0 0x7fed7c06a7a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.896+0000 7fed97a0c700 1 --2- 172.21.15.116:0/2617310908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed9015f9b0 0x7fed901e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.896+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 >> 172.21.15.116:0/2617310908 conn(0x7fed9006c520 msgr2=0x7fed901609f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.897+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 shutdown_connections 2023-11-08T08:18:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:42.897+0000 7fed97a0c700 1 -- 172.21.15.116:0/2617310908 wait complete. 2023-11-08T08:18:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 3 and data pool 4 2023-11-08T08:18:42.918 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.037+0000 7f04d3282700 1 Processor -- start 2023-11-08T08:18:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.037+0000 7f04d3282700 1 -- start start 2023-11-08T08:18:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d3282700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04cc15cc80 0x7f04cc06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d3282700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0b1810 con 0x7f04cc06a270 2023-11-08T08:18:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d3282700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0b1af0 con 0x7f04cc06e3a0 2023-11-08T08:18:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d3282700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04cc0b2490 con 0x7f04cc15cc80 2023-11-08T08:18:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d081d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04cc15cc80 0x7f04cc06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d101e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f04cc06a270 0x7f04cc06a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34244/0 (socket says 172.21.15.116:34244) 2023-11-08T08:18:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d101e700 1 -- 172.21.15.116:0/1240875616 learned_addr learned my addr 172.21.15.116:0/1240875616 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d081d700 1 -- 172.21.15.116:0/1240875616 >> v1:172.21.15.129:6789/0 conn(0x7f04cc06e3a0 legacy=0x7f04cc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d081d700 1 -- 172.21.15.116:0/1240875616 >> v1:172.21.15.116:6789/0 conn(0x7f04cc06a270 legacy=0x7f04cc06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04c3fff700 1 -- 172.21.15.116:0/1240875616 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699280028 0 0) 0x7f04cc0b1810 con 0x7f04cc06a270 2023-11-08T08:18:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.038+0000 7f04d081d700 1 -- 172.21.15.116:0/1240875616 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0b1dd0 con 0x7f04cc15cc80 2023-11-08T08:18:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d081d700 1 --2- 172.21.15.116:0/1240875616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc06de60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009840 tx=0x7f04c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb45303b69b2272a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04c3fff700 1 -- 172.21.15.116:0/1240875616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8011830 con 0x7f04cc15cc80 2023-11-08T08:18:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04c3fff700 1 -- 172.21.15.116:0/1240875616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8011990 con 0x7f04cc15cc80 2023-11-08T08:18:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04c3fff700 1 -- 172.21.15.116:0/1240875616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8011d20 con 0x7f04cc15cc80 2023-11-08T08:18:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 -- 172.21.15.116:0/1240875616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 msgr2=0x7f04cc06de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 --2- 172.21.15.116:0/1240875616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc06de60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009840 tx=0x7f04c8009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 -- 172.21.15.116:0/1240875616 shutdown_connections 2023-11-08T08:18:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 --2- 172.21.15.116:0/1240875616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc06de60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 -- 172.21.15.116:0/1240875616 >> 172.21.15.116:0/1240875616 conn(0x7f04cc06c010 msgr2=0x7f04cc0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 -- 172.21.15.116:0/1240875616 shutdown_connections 2023-11-08T08:18:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.039+0000 7f04d3282700 1 -- 172.21.15.116:0/1240875616 wait complete. 2023-11-08T08:18:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.040+0000 7f04d3282700 1 Processor -- start 2023-11-08T08:18:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.040+0000 7f04d3282700 1 -- start start 2023-11-08T08:18:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d3282700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04cc06a270 0x7f04cc1f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d3282700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04cc06e3a0 0x7f04cc15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d3282700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc1e60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d3282700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc15dee0 con 0x7f04cc15cc80 2023-11-08T08:18:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d3282700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc15e1c0 con 0x7f04cc06e3a0 2023-11-08T08:18:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d3282700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc15e4a0 con 0x7f04cc06a270 2023-11-08T08:18:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d181f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc1e60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d181f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc1e60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38976/0 (socket says 172.21.15.116:38976) 2023-11-08T08:18:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d181f700 1 -- 172.21.15.116:0/1881270863 learned_addr learned my addr 172.21.15.116:0/1881270863 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d181f700 1 -- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04cc06a270 msgr2=0x7f04cc1f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d081d700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04cc06e3a0 0x7f04cc15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d101e700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04cc06a270 0x7f04cc1f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.041+0000 7f04d181f700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04cc06a270 0x7f04cc1f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d181f700 1 -- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04cc06e3a0 msgr2=0x7f04cc15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d181f700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04cc06e3a0 0x7f04cc15ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d181f700 1 -- 172.21.15.116:0/1881270863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80094a0 con 0x7f04cc15cc80 2023-11-08T08:18:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d081d700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04cc06e3a0 0x7f04cc15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d181f700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc1e60b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f04c400cb50 tx=0x7f04c400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4003c00 con 0x7f04cc15cc80 2023-11-08T08:18:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c4003d60 con 0x7f04cc15cc80 2023-11-08T08:18:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc06e7c0 con 0x7f04cc15cc80 2023-11-08T08:18:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c400ab70 con 0x7f04cc15cc80 2023-11-08T08:18:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.042+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc0b20b0 con 0x7f04cc15cc80 2023-11-08T08:18:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.043+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc06e950 con 0x7f04cc15cc80 2023-11-08T08:18:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.043+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04c4010050 con 0x7f04cc15cc80 2023-11-08T08:18:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.043+0000 7f04c1ffb700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04b00682f0 0x7f04b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.044+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f04c4085b70 con 0x7f04cc15cc80 2023-11-08T08:18:43.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.044+0000 7f04d101e700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04b00682f0 0x7f04b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.044+0000 7f04d101e700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04b00682f0 0x7f04b006a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f04bc007fd0 tx=0x7f04bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.048+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04cc06e950 con 0x7f04cc15cc80 2023-11-08T08:18:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.230+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04cc06e950 con 0x7f04cc15cc80 2023-11-08T08:18:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.231+0000 7f04c1ffb700 1 -- 172.21.15.116:0/1881270863 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f04cc06e950 con 0x7f04cc15cc80 2023-11-08T08:18:43.231 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:43.231 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:42.885125+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04b00682f0 msgr2=0x7f04b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04b00682f0 0x7f04b006a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f04bc007fd0 tx=0x7f04bc009040 comp rx=0 tx=0).stop 2023-11-08T08:18:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 msgr2=0x7f04cc1e60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc1e60b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f04c400cb50 tx=0x7f04c400cf10 comp rx=0 tx=0).stop 2023-11-08T08:18:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 shutdown_connections 2023-11-08T08:18:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04cc15cc80 0x7f04cc1e60b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04cc06a270 0x7f04cc1f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04b00682f0 0x7f04b006a7a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 --2- 172.21.15.116:0/1881270863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04cc06e3a0 0x7f04cc15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 >> 172.21.15.116:0/1881270863 conn(0x7f04cc06c010 msgr2=0x7f04cc0b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 shutdown_connections 2023-11-08T08:18:43.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.233+0000 7f04d3282700 1 -- 172.21.15.116:0/1881270863 wait complete. 2023-11-08T08:18:43.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.365+0000 7f781eafd700 1 Processor -- start 2023-11-08T08:18:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.365+0000 7f781eafd700 1 -- start start 2023-11-08T08:18:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.365+0000 7f781eafd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f781806a270 0x7f781806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781eafd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f781806de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781eafd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781806ab80 con 0x7f781806e3a0 2023-11-08T08:18:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781eafd700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78180b2490 con 0x7f781806a270 2023-11-08T08:18:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781eafd700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78180b1810 con 0x7f781815cc80 2023-11-08T08:18:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f781806a270 0x7f781806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f781806a270 0x7f781806a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38978/0 (socket says 172.21.15.116:38978) 2023-11-08T08:18:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 -- 172.21.15.116:0/784344191 learned_addr learned my addr 172.21.15.116:0/784344191 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f7817fff700 1 --2- 172.21.15.116:0/784344191 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f781806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 -- 172.21.15.116:0/784344191 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 msgr2=0x7f781806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 --2- 172.21.15.116:0/784344191 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f781806de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 -- 172.21.15.116:0/784344191 >> v1:172.21.15.129:6789/0 conn(0x7f781806e3a0 legacy=0x7f781806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 -- 172.21.15.116:0/784344191 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180b1af0 con 0x7f781806a270 2023-11-08T08:18:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f781c899700 1 --2- 172.21.15.116:0/784344191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806a270 0x7f781806a640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f780c0098f0 tx=0x7f780c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0b52a8f0952ef4 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f78177fe700 1 -- 172.21.15.116:0/784344191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c004230 con 0x7f781806a270 2023-11-08T08:18:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f78177fe700 1 -- 172.21.15.116:0/784344191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c004390 con 0x7f781806a270 2023-11-08T08:18:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.366+0000 7f78177fe700 1 -- 172.21.15.116:0/784344191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c0036a0 con 0x7f781806a270 2023-11-08T08:18:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 -- 172.21.15.116:0/784344191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806a270 msgr2=0x7f781806a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 --2- 172.21.15.116:0/784344191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806a270 0x7f781806a640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f780c0098f0 tx=0x7f780c009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 -- 172.21.15.116:0/784344191 shutdown_connections 2023-11-08T08:18:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 --2- 172.21.15.116:0/784344191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806a270 0x7f781806a640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 --2- 172.21.15.116:0/784344191 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f781806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 -- 172.21.15.116:0/784344191 >> 172.21.15.116:0/784344191 conn(0x7f781806c010 msgr2=0x7f78180cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 -- 172.21.15.116:0/784344191 shutdown_connections 2023-11-08T08:18:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.367+0000 7f781eafd700 1 -- 172.21.15.116:0/784344191 wait complete. 2023-11-08T08:18:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.368+0000 7f781eafd700 1 Processor -- start 2023-11-08T08:18:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.368+0000 7f781eafd700 1 -- start start 2023-11-08T08:18:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f781806a270 0x7f7818153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 0x7f78180b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f78180b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78180c6dc0 con 0x7f781806e3a0 2023-11-08T08:18:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781815ea20 con 0x7f781806a270 2023-11-08T08:18:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781815ed00 con 0x7f781815cc80 2023-11-08T08:18:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 0x7f78180b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 0x7f78180b35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38986/0 (socket says 172.21.15.116:38986) 2023-11-08T08:18:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 -- 172.21.15.116:0/1328703427 learned_addr learned my addr 172.21.15.116:0/1328703427 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781c899700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f781806a270 0x7f7818153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781d09a700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f78180b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 -- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 msgr2=0x7f78180b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f78180b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 -- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f781806a270 msgr2=0x7f7818153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f781806a270 0x7f7818153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 -- 172.21.15.116:0/1328703427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c009550 con 0x7f781806e3a0 2023-11-08T08:18:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f7817fff700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 0x7f78180b35d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f780400d800 tx=0x7f780400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78040125f0 con 0x7f781806e3a0 2023-11-08T08:18:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804012750 con 0x7f781806e3a0 2023-11-08T08:18:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78180b20b0 con 0x7f781806e3a0 2023-11-08T08:18:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.369+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804012b00 con 0x7f781806e3a0 2023-11-08T08:18:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.370+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78180b1dd0 con 0x7f781806e3a0 2023-11-08T08:18:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.370+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781806e7e0 con 0x7f781806e3a0 2023-11-08T08:18:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.371+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7804012c60 con 0x7f781806e3a0 2023-11-08T08:18:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.371+0000 7f78157fa700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f77fc0682f0 0x7f77fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.371+0000 7f781c899700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f77fc0682f0 0x7f77fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.371+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7804085f50 con 0x7f781806e3a0 2023-11-08T08:18:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.372+0000 7f781c899700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f77fc0682f0 0x7f77fc06a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f780c009130 tx=0x7f780c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.376+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78040526c0 con 0x7f781806e3a0 2023-11-08T08:18:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.557+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78180743f0 con 0x7f781806e3a0 2023-11-08T08:18:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.559+0000 7f78157fa700 1 -- 172.21.15.116:0/1328703427 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f7804055ce0 con 0x7f781806e3a0 2023-11-08T08:18:43.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:43.560 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:42.885125+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:43.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f77fc0682f0 msgr2=0x7f77fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f77fc0682f0 0x7f77fc06a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f780c009130 tx=0x7f780c008030 comp rx=0 tx=0).stop 2023-11-08T08:18:43.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 msgr2=0x7f78180b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 0x7f78180b35d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f780400d800 tx=0x7f780400db10 comp rx=0 tx=0).stop 2023-11-08T08:18:43.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 shutdown_connections 2023-11-08T08:18:43.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f781815cc80 0x7f78180b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f77fc0682f0 0x7f77fc06a7a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f781806a270 0x7f7818153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 --2- 172.21.15.116:0/1328703427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781806e3a0 0x7f78180b35d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 >> 172.21.15.116:0/1328703427 conn(0x7f781806c010 msgr2=0x7f78180ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:43.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 shutdown_connections 2023-11-08T08:18:43.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.561+0000 7f781eafd700 1 -- 172.21.15.116:0/1328703427 wait complete. 2023-11-08T08:18:43.573 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-08T08:18:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.696+0000 7f12312e2700 1 Processor -- start 2023-11-08T08:18:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.696+0000 7f12312e2700 1 -- start start 2023-11-08T08:18:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f12312e2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f122c15c8c0 0x7f122c160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f12312e2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f122c0b5dc0 0x7f122c06ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f12312e2700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0d2d30 con 0x7f122c06b050 2023-11-08T08:18:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f12312e2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122c0d0b40 con 0x7f122c15c8c0 2023-11-08T08:18:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f12312e2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122c06b490 con 0x7f122c0b5dc0 2023-11-08T08:18:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122affd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f122c15c8c0 0x7f122c160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122b7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f122c06b050 0x7f122c0d0fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34254/0 (socket says 172.21.15.116:34254) 2023-11-08T08:18:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122b7fe700 1 -- 172.21.15.116:0/1719758540 learned_addr learned my addr 172.21.15.116:0/1719758540 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122affd700 1 -- 172.21.15.116:0/1719758540 >> v2:172.21.15.129:3300/0 conn(0x7f122c0b5dc0 msgr2=0x7f122c06ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122a7fc700 1 --2- 172.21.15.116:0/1719758540 >> v2:172.21.15.129:3300/0 conn(0x7f122c0b5dc0 0x7f122c06ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122affd700 1 --2- 172.21.15.116:0/1719758540 >> v2:172.21.15.129:3300/0 conn(0x7f122c0b5dc0 0x7f122c06ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122affd700 1 -- 172.21.15.116:0/1719758540 >> v1:172.21.15.116:6789/0 conn(0x7f122c06b050 legacy=0x7f122c0d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f1229ffb700 1 -- 172.21.15.116:0/1719758540 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925019001 0 0) 0x7f1224001490 con 0x7f122c06b050 2023-11-08T08:18:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122affd700 1 -- 172.21.15.116:0/1719758540 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c1611f0 con 0x7f122c15c8c0 2023-11-08T08:18:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122a7fc700 1 --2- 172.21.15.116:0/1719758540 >> v2:172.21.15.129:3300/0 conn(0x7f122c0b5dc0 0x7f122c06ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f122affd700 1 --2- 172.21.15.116:0/1719758540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c15c8c0 0x7f122c160cb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f122c0d24c0 tx=0x7f1220009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faed6f1b888103f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.697+0000 7f1229ffb700 1 -- 172.21.15.116:0/1719758540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220011830 con 0x7f122c15c8c0 2023-11-08T08:18:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f1229ffb700 1 -- 172.21.15.116:0/1719758540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1220011990 con 0x7f122c15c8c0 2023-11-08T08:18:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f1229ffb700 1 -- 172.21.15.116:0/1719758540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220011d20 con 0x7f122c15c8c0 2023-11-08T08:18:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 -- 172.21.15.116:0/1719758540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c15c8c0 msgr2=0x7f122c160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 --2- 172.21.15.116:0/1719758540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c15c8c0 0x7f122c160cb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f122c0d24c0 tx=0x7f1220009b20 comp rx=0 tx=0).stop 2023-11-08T08:18:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 -- 172.21.15.116:0/1719758540 shutdown_connections 2023-11-08T08:18:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 --2- 172.21.15.116:0/1719758540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c15c8c0 0x7f122c160cb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 --2- 172.21.15.116:0/1719758540 >> v2:172.21.15.129:3300/0 conn(0x7f122c0b5dc0 0x7f122c06ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 -- 172.21.15.116:0/1719758540 >> 172.21.15.116:0/1719758540 conn(0x7f122c06c520 msgr2=0x7f122c06c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 -- 172.21.15.116:0/1719758540 shutdown_connections 2023-11-08T08:18:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.698+0000 7f12312e2700 1 -- 172.21.15.116:0/1719758540 wait complete. 2023-11-08T08:18:43.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.699+0000 7f12312e2700 1 Processor -- start 2023-11-08T08:18:43.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.699+0000 7f12312e2700 1 -- start start 2023-11-08T08:18:43.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f12312e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 0x7f122c0b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f12312e2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f122c0b5dc0 0x7f122c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f12312e2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f122c15c8c0 0x7f122c0ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f12312e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0c7c20 con 0x7f122c06b050 2023-11-08T08:18:43.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f12312e2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0c8c60 con 0x7f122c15c8c0 2023-11-08T08:18:43.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f12312e2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0ca510 con 0x7f122c0b5dc0 2023-11-08T08:18:43.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 0x7f122c0b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 0x7f122c0b81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39010/0 (socket says 172.21.15.116:39010) 2023-11-08T08:18:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 -- 172.21.15.116:0/961093374 learned_addr learned my addr 172.21.15.116:0/961093374 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 -- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f122c0b5dc0 msgr2=0x7f122c15d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f122c0b5dc0 0x7f122c15d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 -- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f122c15c8c0 msgr2=0x7f122c0ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122b7fe700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f122c15c8c0 0x7f122c0ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f122c15c8c0 0x7f122c0ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:43.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.700+0000 7f122affd700 1 -- 172.21.15.116:0/961093374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200094a0 con 0x7f122c06b050 2023-11-08T08:18:43.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.701+0000 7f122affd700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 0x7f122c0b81c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f12200090b0 tx=0x7f1220004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.701+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12200036a0 con 0x7f122c06b050 2023-11-08T08:18:43.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.701+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122001e070 con 0x7f122c06b050 2023-11-08T08:18:43.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.701+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c0cd760 con 0x7f122c06b050 2023-11-08T08:18:43.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.701+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220003a60 con 0x7f122c06b050 2023-11-08T08:18:43.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.701+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c0d2a50 con 0x7f122c06b050 2023-11-08T08:18:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.702+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122c0cda40 con 0x7f122c06b050 2023-11-08T08:18:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.702+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f122000f050 con 0x7f122c06b050 2023-11-08T08:18:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.702+0000 7f12137fe700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f12180682f0 0x7f121806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.702+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1220086da0 con 0x7f122c06b050 2023-11-08T08:18:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.703+0000 7f122a7fc700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f12180682f0 0x7f121806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:43.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.703+0000 7f122a7fc700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f12180682f0 0x7f121806a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f121c006fd0 tx=0x7f121c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:43.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.707+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1220053600 con 0x7f122c06b050 2023-11-08T08:18:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:43.932+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f122c067d70 con 0x7f122c06b050 2023-11-08T08:18:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.895+0000 7f12137fe700 1 -- 172.21.15.116:0/961093374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1220015020 con 0x7f122c06b050 2023-11-08T08:18:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f12180682f0 msgr2=0x7f121806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f12180682f0 0x7f121806a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f121c006fd0 tx=0x7f121c008040 comp rx=0 tx=0).stop 2023-11-08T08:18:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 msgr2=0x7f122c0b81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 0x7f122c0b81c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f12200090b0 tx=0x7f1220004750 comp rx=0 tx=0).stop 2023-11-08T08:18:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 shutdown_connections 2023-11-08T08:18:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f12180682f0 0x7f121806a7a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f122c15c8c0 0x7f122c0ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f122c06b050 0x7f122c0b81c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 --2- 172.21.15.116:0/961093374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f122c0b5dc0 0x7f122c15d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.897+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 >> 172.21.15.116:0/961093374 conn(0x7f122c06c520 msgr2=0x7f122c157690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.898+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 shutdown_connections 2023-11-08T08:18:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:44.898+0000 7f12312e2700 1 -- 172.21.15.116:0/961093374 wait complete. 2023-11-08T08:18:44.912 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-08T08:18:44.912 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-08T08:18:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 Processor -- start 2023-11-08T08:18:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 -- start start 2023-11-08T08:18:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f45600ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f456006a980 0x7f4560071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456015cc10 con 0x7f4560071890 2023-11-08T08:18:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456006c9b0 con 0x7f456006a980 2023-11-08T08:18:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.041+0000 7f4568579700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456006cc90 con 0x7f45600cf710 2023-11-08T08:18:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f456006a980 0x7f4560071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f456006a980 0x7f4560071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39024/0 (socket says 172.21.15.116:39024) 2023-11-08T08:18:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 -- 172.21.15.116:0/2270344986 learned_addr learned my addr 172.21.15.116:0/2270344986 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 -- 172.21.15.116:0/2270344986 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 msgr2=0x7f45600ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 --2- 172.21.15.116:0/2270344986 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f45600ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 -- 172.21.15.116:0/2270344986 >> v1:172.21.15.129:6789/0 conn(0x7f4560071890 legacy=0x7f456006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:45.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 -- 172.21.15.116:0/2270344986 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456006cf70 con 0x7f456006a980 2023-11-08T08:18:45.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4566b16700 1 --2- 172.21.15.116:0/2270344986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f4560071220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4558009e50 tx=0x7f455800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba64388829e8fd57 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:45.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4565313700 1 -- 172.21.15.116:0/2270344986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455800e070 con 0x7f456006a980 2023-11-08T08:18:45.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4565313700 1 -- 172.21.15.116:0/2270344986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558003a20 con 0x7f456006a980 2023-11-08T08:18:45.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.042+0000 7f4565313700 1 -- 172.21.15.116:0/2270344986 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558003dd0 con 0x7f456006a980 2023-11-08T08:18:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 -- 172.21.15.116:0/2270344986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 msgr2=0x7f4560071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 --2- 172.21.15.116:0/2270344986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f4560071220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4558009e50 tx=0x7f455800b330 comp rx=0 tx=0).stop 2023-11-08T08:18:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 -- 172.21.15.116:0/2270344986 shutdown_connections 2023-11-08T08:18:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 --2- 172.21.15.116:0/2270344986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f4560071220 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 --2- 172.21.15.116:0/2270344986 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f45600ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 -- 172.21.15.116:0/2270344986 >> 172.21.15.116:0/2270344986 conn(0x7f456006c010 msgr2=0x7f45600cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 -- 172.21.15.116:0/2270344986 shutdown_connections 2023-11-08T08:18:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.043+0000 7f4568579700 1 -- 172.21.15.116:0/2270344986 wait complete. 2023-11-08T08:18:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.044+0000 7f4568579700 1 Processor -- start 2023-11-08T08:18:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.044+0000 7f4568579700 1 -- start start 2023-11-08T08:18:45.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f456015e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:45.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4560071890 0x7f45600b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:45.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f4560157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:45.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456015e970 con 0x7f456006a980 2023-11-08T08:18:45.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600b3db0 con 0x7f4560071890 2023-11-08T08:18:45.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45600b2f40 con 0x7f45600cf710 2023-11-08T08:18:45.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f456015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:45.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f456015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39040/0 (socket says 172.21.15.116:39040) 2023-11-08T08:18:45.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 -- 172.21.15.116:0/181150944 learned_addr learned my addr 172.21.15.116:0/181150944 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:45.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566b16700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f4560157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:45.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4565b14700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4560071890 0x7f45600b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:45.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 -- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 msgr2=0x7f4560157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:45.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f4560157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 -- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4560071890 msgr2=0x7f45600b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:45.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4560071890 0x7f45600b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 -- 172.21.15.116:0/181150944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558009b30 con 0x7f456006a980 2023-11-08T08:18:45.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4566315700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f456015e430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f455000b4a0 tx=0x7f455000b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:45.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550011830 con 0x7f456006a980 2023-11-08T08:18:45.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4550011990 con 0x7f456006a980 2023-11-08T08:18:45.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550011d40 con 0x7f456006a980 2023-11-08T08:18:45.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456006d530 con 0x7f456006a980 2023-11-08T08:18:45.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.045+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456006d250 con 0x7f456006a980 2023-11-08T08:18:45.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.046+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560160480 con 0x7f456006a980 2023-11-08T08:18:45.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.047+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4550021070 con 0x7f456006a980 2023-11-08T08:18:45.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.047+0000 7f4556ffd700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f45440682f0 0x7f454406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:45.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.047+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4550085a00 con 0x7f456006a980 2023-11-08T08:18:45.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.047+0000 7f4565b14700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f45440682f0 0x7f454406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:45.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.047+0000 7f4565b14700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f45440682f0 0x7f454406a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f456015acc0 tx=0x7f455c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:45.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.051+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45500521e0 con 0x7f456006a980 2023-11-08T08:18:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.276+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4560074440 con 0x7f456006a980 2023-11-08T08:18:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.901+0000 7f4556ffd700 1 -- 172.21.15.116:0/181150944 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f45500150c0 con 0x7f456006a980 2023-11-08T08:18:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f45440682f0 msgr2=0x7f454406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f45440682f0 0x7f454406a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f456015acc0 tx=0x7f455c009040 comp rx=0 tx=0).stop 2023-11-08T08:18:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 msgr2=0x7f456015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f456015e430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f455000b4a0 tx=0x7f455000b7b0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 shutdown_connections 2023-11-08T08:18:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45600cf710 0x7f4560157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f456006a980 0x7f456015e430 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f45440682f0 0x7f454406a7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 --2- 172.21.15.116:0/181150944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4560071890 0x7f45600b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 >> 172.21.15.116:0/181150944 conn(0x7f456006c010 msgr2=0x7f45600b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 shutdown_connections 2023-11-08T08:18:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:45.904+0000 7f4568579700 1 -- 172.21.15.116:0/181150944 wait complete. 2023-11-08T08:18:45.918 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-08T08:18:45.918 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:18:46.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.045+0000 7f6fccf77700 1 Processor -- start 2023-11-08T08:18:46.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.045+0000 7f6fccf77700 1 -- start start 2023-11-08T08:18:46.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fccf77700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc806b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fccf77700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6fc8072180 0x7f6fc80aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fccf77700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc80b43f0 con 0x7f6fc80af070 2023-11-08T08:18:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fccf77700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc80b46d0 con 0x7f6fc8072180 2023-11-08T08:18:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fccf77700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc80b4d00 con 0x7f6fc80b0880 2023-11-08T08:18:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fc659c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc806b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fc5d9b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6fc8072180 0x7f6fc80aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fc659c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc806b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52178/0 (socket says 172.21.15.116:52178) 2023-11-08T08:18:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.046+0000 7f6fc659c700 1 -- 172.21.15.116:0/3267840127 learned_addr learned my addr 172.21.15.116:0/3267840127 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 msgr2=0x7f6fc806b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:46.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc5d9b700 1 --2- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc806b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3267840127 >> v1:172.21.15.129:6789/0 conn(0x7f6fc80af070 legacy=0x7f6fc80af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3267840127 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc80b4fe0 con 0x7f6fc8072180 2023-11-08T08:18:46.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc659c700 1 --2- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc806b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:46.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc5d9b700 1 --2- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc80aeb30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009840 tx=0x7f6fb8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b763b9d00aebef6d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc559a700 1 -- 172.21.15.116:0/3267840127 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8011830 con 0x7f6fc8072180 2023-11-08T08:18:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc559a700 1 -- 172.21.15.116:0/3267840127 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb8011990 con 0x7f6fc8072180 2023-11-08T08:18:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fc559a700 1 -- 172.21.15.116:0/3267840127 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8011d20 con 0x7f6fc8072180 2023-11-08T08:18:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fccf77700 1 -- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 msgr2=0x7f6fc80aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.047+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc80aeb30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009840 tx=0x7f6fb8009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.048+0000 7f6fccf77700 1 -- 172.21.15.116:0/3267840127 shutdown_connections 2023-11-08T08:18:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.048+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc806b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.048+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3267840127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc80aeb30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.048+0000 7f6fccf77700 1 -- 172.21.15.116:0/3267840127 >> 172.21.15.116:0/3267840127 conn(0x7f6fc80b2fc0 msgr2=0x7f6fc8071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.048+0000 7f6fccf77700 1 -- 172.21.15.116:0/3267840127 shutdown_connections 2023-11-08T08:18:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.048+0000 7f6fccf77700 1 -- 172.21.15.116:0/3267840127 wait complete. 2023-11-08T08:18:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.049+0000 7f6fccf77700 1 Processor -- start 2023-11-08T08:18:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.049+0000 7f6fccf77700 1 -- start start 2023-11-08T08:18:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc8159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 0x7f6fc8098890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8158c80 con 0x7f6fc80af070 2023-11-08T08:18:46.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8159650 con 0x7f6fc8072180 2023-11-08T08:18:46.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8158350 con 0x7f6fc80b0880 2023-11-08T08:18:46.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 0x7f6fc8098890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 0x7f6fc8098890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39046/0 (socket says 172.21.15.116:39046) 2023-11-08T08:18:46.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3085577822 learned_addr learned my addr 172.21.15.116:0/3085577822 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc659c700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc8159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc6d9d700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 msgr2=0x7f6fc815a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc815a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 msgr2=0x7f6fc8159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc8159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 -- 172.21.15.116:0/3085577822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc009550 con 0x7f6fc80af070 2023-11-08T08:18:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fc5d9b700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 0x7f6fc8098890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009fd0 tx=0x7f6fb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb80036a0 con 0x7f6fc80af070 2023-11-08T08:18:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb801e070 con 0x7f6fc80af070 2023-11-08T08:18:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb80094a0 con 0x7f6fc80af070 2023-11-08T08:18:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8003a40 con 0x7f6fc80af070 2023-11-08T08:18:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.050+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc80b49b0 con 0x7f6fc80af070 2023-11-08T08:18:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.051+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc80725f0 con 0x7f6fc80af070 2023-11-08T08:18:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.052+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6fb800f050 con 0x7f6fc80af070 2023-11-08T08:18:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.052+0000 7f6fb6ffd700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fa80682f0 0x7f6fa806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.052+0000 7f6fc659c700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fa80682f0 0x7f6fa806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.052+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6fb8086ba0 con 0x7f6fc80af070 2023-11-08T08:18:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.053+0000 7f6fc659c700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fa80682f0 0x7f6fa806a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc006d30 tx=0x7f6fbc0092c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.057+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6fb8053400 con 0x7f6fc80af070 2023-11-08T08:18:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.277+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fc80b0c50 con 0x7f6fc80af070 2023-11-08T08:18:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.277+0000 7f6fb6ffd700 1 -- 172.21.15.116:0/3085577822 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4654 (secure 0 0 0) 0x7f6fb8015020 con 0x7f6fc80af070 2023-11-08T08:18:46.278 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:46.278 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:18:42.885473+0000","modified":"2023-11-08T08:18:45.898738+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-08T08:18:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fa80682f0 msgr2=0x7f6fa806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fa80682f0 0x7f6fa806a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc006d30 tx=0x7f6fbc0092c0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 msgr2=0x7f6fc8098890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 0x7f6fc8098890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009fd0 tx=0x7f6fb8004750 comp rx=0 tx=0).stop 2023-11-08T08:18:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 shutdown_connections 2023-11-08T08:18:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fc80b0880 0x7f6fc815a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fa80682f0 0x7f6fa806a7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc8072180 0x7f6fc8159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 --2- 172.21.15.116:0/3085577822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc80af070 0x7f6fc8098890 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 >> 172.21.15.116:0/3085577822 conn(0x7f6fc80b2fc0 msgr2=0x7f6fc815c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 shutdown_connections 2023-11-08T08:18:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.279+0000 7f6fccf77700 1 -- 172.21.15.116:0/3085577822 wait complete. 2023-11-08T08:18:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-08T08:18:46.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 Processor -- start 2023-11-08T08:18:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 -- start start 2023-11-08T08:18:46.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f78c8142db0 0x7f78c813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f78c80b1770 0x7f78c80b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c815e010 con 0x7f78c80ce040 2023-11-08T08:18:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c813fc90 con 0x7f78c80b1770 2023-11-08T08:18:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.412+0000 7f78ce553700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c80b20d0 con 0x7f78c8142db0 2023-11-08T08:18:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78ccaf0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f78c80b1770 0x7f78c80b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c7fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f78c80ce040 0x7f78c815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34260/0 (socket says 172.21.15.116:34260) 2023-11-08T08:18:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c7fff700 1 -- 172.21.15.116:0/704515136 learned_addr learned my addr 172.21.15.116:0/704515136 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c77fe700 1 --2- 172.21.15.116:0/704515136 >> v2:172.21.15.129:3300/0 conn(0x7f78c8142db0 0x7f78c813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78ccaf0700 1 -- 172.21.15.116:0/704515136 >> v2:172.21.15.129:3300/0 conn(0x7f78c8142db0 msgr2=0x7f78c813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78ccaf0700 1 --2- 172.21.15.116:0/704515136 >> v2:172.21.15.129:3300/0 conn(0x7f78c8142db0 0x7f78c813ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78ccaf0700 1 -- 172.21.15.116:0/704515136 >> v1:172.21.15.116:6789/0 conn(0x7f78c80ce040 legacy=0x7f78c815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c6ffd700 1 -- 172.21.15.116:0/704515136 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040804067 0 0) 0x7f78b8001110 con 0x7f78c80ce040 2023-11-08T08:18:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78ccaf0700 1 -- 172.21.15.116:0/704515136 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c813f6d0 con 0x7f78c80b1770 2023-11-08T08:18:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78ccaf0700 1 --2- 172.21.15.116:0/704515136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c80b1770 0x7f78c80b1b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f78c80cfc20 tx=0x7f78c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c644a3a2734cec server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c6ffd700 1 -- 172.21.15.116:0/704515136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c000e070 con 0x7f78c80b1770 2023-11-08T08:18:46.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c6ffd700 1 -- 172.21.15.116:0/704515136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c000bb20 con 0x7f78c80b1770 2023-11-08T08:18:46.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.413+0000 7f78c6ffd700 1 -- 172.21.15.116:0/704515136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0012430 con 0x7f78c80b1770 2023-11-08T08:18:46.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 -- 172.21.15.116:0/704515136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c80b1770 msgr2=0x7f78c80b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 --2- 172.21.15.116:0/704515136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c80b1770 0x7f78c80b1b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f78c80cfc20 tx=0x7f78c000b330 comp rx=0 tx=0).stop 2023-11-08T08:18:46.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 -- 172.21.15.116:0/704515136 shutdown_connections 2023-11-08T08:18:46.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 --2- 172.21.15.116:0/704515136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c80b1770 0x7f78c80b1b90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 --2- 172.21.15.116:0/704515136 >> v2:172.21.15.129:3300/0 conn(0x7f78c8142db0 0x7f78c813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 -- 172.21.15.116:0/704515136 >> 172.21.15.116:0/704515136 conn(0x7f78c806c010 msgr2=0x7f78c80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:46.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 -- 172.21.15.116:0/704515136 shutdown_connections 2023-11-08T08:18:46.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.414+0000 7f78ce553700 1 -- 172.21.15.116:0/704515136 wait complete. 2023-11-08T08:18:46.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.415+0000 7f78ce553700 1 Processor -- start 2023-11-08T08:18:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.415+0000 7f78ce553700 1 -- start start 2023-11-08T08:18:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c80b1770 0x7f78c813f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f78c80ce040 0x7f78c8154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 0x7f78c80c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c8155490 con 0x7f78c8142db0 2023-11-08T08:18:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80c5a20 con 0x7f78c80b1770 2023-11-08T08:18:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80c5c90 con 0x7f78c80ce040 2023-11-08T08:18:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 0x7f78c80c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 0x7f78c80c37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39066/0 (socket says 172.21.15.116:39066) 2023-11-08T08:18:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 -- 172.21.15.116:0/3854573645 learned_addr learned my addr 172.21.15.116:0/3854573645 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78c7fff700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c80b1770 0x7f78c813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78c77fe700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f78c80ce040 0x7f78c8154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 -- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f78c80ce040 msgr2=0x7f78c8154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f78c80ce040 0x7f78c8154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 -- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c80b1770 msgr2=0x7f78c813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c80b1770 0x7f78c813f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 -- 172.21.15.116:0/3854573645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c0009a70 con 0x7f78c8142db0 2023-11-08T08:18:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ccaf0700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 0x7f78c80c37e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f78c0014040 tx=0x7f78c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0012430 con 0x7f78c8142db0 2023-11-08T08:18:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0020070 con 0x7f78c8142db0 2023-11-08T08:18:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0004280 con 0x7f78c8142db0 2023-11-08T08:18:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c815d100 con 0x7f78c8142db0 2023-11-08T08:18:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.416+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c813f9b0 con 0x7f78c8142db0 2023-11-08T08:18:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.417+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c815d3e0 con 0x7f78c8142db0 2023-11-08T08:18:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.418+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f78c002e070 con 0x7f78c8142db0 2023-11-08T08:18:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.418+0000 7f78c4ff9700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78b40682f0 0x7f78b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.418+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f78c0022070 con 0x7f78c8142db0 2023-11-08T08:18:46.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.418+0000 7f78c7fff700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78b40682f0 0x7f78b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.419+0000 7f78c7fff700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78b40682f0 0x7f78b406a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78b8000f60 tx=0x7f78b8007c20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.422+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78c00540d0 con 0x7f78c8142db0 2023-11-08T08:18:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.603+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78c80743f0 con 0x7f78c8142db0 2023-11-08T08:18:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.605+0000 7f78c4ff9700 1 -- 172.21.15.116:0/3854573645 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f78c00576f0 con 0x7f78c8142db0 2023-11-08T08:18:46.605 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:46.605 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.606+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78b40682f0 msgr2=0x7f78b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.606+0000 7f78ce553700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78b40682f0 0x7f78b406a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78b8000f60 tx=0x7f78b8007c20 comp rx=0 tx=0).stop 2023-11-08T08:18:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.606+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 msgr2=0x7f78c80c37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.606+0000 7f78ce553700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 0x7f78c80c37e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f78c0014040 tx=0x7f78c0004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 shutdown_connections 2023-11-08T08:18:46.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c8142db0 0x7f78c80c37e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78b40682f0 0x7f78b406a7a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c80b1770 0x7f78c813f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 --2- 172.21.15.116:0/3854573645 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f78c80ce040 0x7f78c8154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 >> 172.21.15.116:0/3854573645 conn(0x7f78c806c010 msgr2=0x7f78c806db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:46.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 shutdown_connections 2023-11-08T08:18:46.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.607+0000 7f78ce553700 1 -- 172.21.15.116:0/3854573645 wait complete. 2023-11-08T08:18:46.619 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-08T08:18:46.620 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:18:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.738+0000 7f23a32f6700 1 Processor -- start 2023-11-08T08:18:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a32f6700 1 -- start start 2023-11-08T08:18:46.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a32f6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f239c0ced70 0x7f239c0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a32f6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f239c06feb0 0x7f239c0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a32f6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0b4fc0 con 0x7f239c072420 2023-11-08T08:18:46.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a32f6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239c159c30 con 0x7f239c06feb0 2023-11-08T08:18:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a32f6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239c0b05e0 con 0x7f239c0ced70 2023-11-08T08:18:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a0891700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f239c06feb0 0x7f239c0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.739+0000 7f23a1893700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f239c072420 0x7f239c06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34274/0 (socket says 172.21.15.116:34274) 2023-11-08T08:18:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a1893700 1 -- 172.21.15.116:0/2656186651 learned_addr learned my addr 172.21.15.116:0/2656186651 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a1092700 1 --2- 172.21.15.116:0/2656186651 >> v2:172.21.15.129:3300/0 conn(0x7f239c0ced70 0x7f239c0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a0891700 1 -- 172.21.15.116:0/2656186651 >> v2:172.21.15.129:3300/0 conn(0x7f239c0ced70 msgr2=0x7f239c0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a0891700 1 --2- 172.21.15.116:0/2656186651 >> v2:172.21.15.129:3300/0 conn(0x7f239c0ced70 0x7f239c0b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a0891700 1 -- 172.21.15.116:0/2656186651 >> v1:172.21.15.116:6789/0 conn(0x7f239c072420 legacy=0x7f239c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a0891700 1 -- 172.21.15.116:0/2656186651 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0b08c0 con 0x7f239c06feb0 2023-11-08T08:18:46.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f239bfff700 1 -- 172.21.15.116:0/2656186651 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069071991 0 0) 0x7f239c0b4fc0 con 0x7f239c072420 2023-11-08T08:18:46.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f23a0891700 1 --2- 172.21.15.116:0/2656186651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c06feb0 0x7f239c0b4930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f239c06eee0 tx=0x7f2390009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1574e31383fcf5cc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f239bfff700 1 -- 172.21.15.116:0/2656186651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390011830 con 0x7f239c06feb0 2023-11-08T08:18:46.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f239bfff700 1 -- 172.21.15.116:0/2656186651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2390011990 con 0x7f239c06feb0 2023-11-08T08:18:46.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.740+0000 7f239bfff700 1 -- 172.21.15.116:0/2656186651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390011d20 con 0x7f239c06feb0 2023-11-08T08:18:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 -- 172.21.15.116:0/2656186651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c06feb0 msgr2=0x7f239c0b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 --2- 172.21.15.116:0/2656186651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c06feb0 0x7f239c0b4930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f239c06eee0 tx=0x7f2390009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 -- 172.21.15.116:0/2656186651 shutdown_connections 2023-11-08T08:18:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 --2- 172.21.15.116:0/2656186651 >> v2:172.21.15.129:3300/0 conn(0x7f239c0ced70 0x7f239c0b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 --2- 172.21.15.116:0/2656186651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c06feb0 0x7f239c0b4930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 -- 172.21.15.116:0/2656186651 >> 172.21.15.116:0/2656186651 conn(0x7f239c0b2fc0 msgr2=0x7f239c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 -- 172.21.15.116:0/2656186651 shutdown_connections 2023-11-08T08:18:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.741+0000 7f23a32f6700 1 -- 172.21.15.116:0/2656186651 wait complete. 2023-11-08T08:18:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.742+0000 7f23a32f6700 1 Processor -- start 2023-11-08T08:18:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.742+0000 7f23a32f6700 1 -- start start 2023-11-08T08:18:46.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a32f6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06feb0 0x7f239c0992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a32f6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 0x7f239c153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a32f6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f239c0ced70 0x7f239c1522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a32f6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0c6360 con 0x7f239c072420 2023-11-08T08:18:46.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a32f6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0c5da0 con 0x7f239c06feb0 2023-11-08T08:18:46.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a32f6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0c6080 con 0x7f239c0ced70 2023-11-08T08:18:46.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 0x7f239c153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 0x7f239c153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39086/0 (socket says 172.21.15.116:39086) 2023-11-08T08:18:46.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 -- 172.21.15.116:0/4187086359 learned_addr learned my addr 172.21.15.116:0/4187086359 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:46.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 -- 172.21.15.116:0/4187086359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f239c0ced70 msgr2=0x7f239c1522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f239c0ced70 0x7f239c1522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 -- 172.21.15.116:0/4187086359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06feb0 msgr2=0x7f239c0992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06feb0 0x7f239c0992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.743+0000 7f23a0891700 1 -- 172.21.15.116:0/4187086359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23900094a0 con 0x7f239c072420 2023-11-08T08:18:46.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.744+0000 7f23a0891700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 0x7f239c153480 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f23900097c0 tx=0x7f2390004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.744+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900036a0 con 0x7f239c072420 2023-11-08T08:18:46.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.744+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239001e070 con 0x7f239c072420 2023-11-08T08:18:46.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.744+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c0b5260 con 0x7f239c072420 2023-11-08T08:18:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.744+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390003a20 con 0x7f239c072420 2023-11-08T08:18:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.744+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c0b0ba0 con 0x7f239c072420 2023-11-08T08:18:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.745+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c0ce1e0 con 0x7f239c072420 2023-11-08T08:18:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.745+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f239000f050 con 0x7f239c072420 2023-11-08T08:18:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.746+0000 7f2399ffb700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2388068320 0x7f238806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.746+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2390086c80 con 0x7f239c072420 2023-11-08T08:18:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.746+0000 7f23a1092700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2388068320 0x7f238806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.746+0000 7f23a1092700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2388068320 0x7f238806a7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2394006b20 tx=0x7f2394006ab0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.750+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23900533f0 con 0x7f239c072420 2023-11-08T08:18:46.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.928+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f239c1f1d80 con 0x7f2388068320 2023-11-08T08:18:46.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.936+0000 7f2399ffb700 1 -- 172.21.15.116:0/4187086359 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286927 (secure 0 0 0) 0x7f239c1f1d80 con 0x7f2388068320 2023-11-08T08:18:46.940 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2388068320 msgr2=0x7f238806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2388068320 0x7f238806a7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2394006b20 tx=0x7f2394006ab0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 msgr2=0x7f239c153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 0x7f239c153480 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f23900097c0 tx=0x7f2390004750 comp rx=0 tx=0).stop 2023-11-08T08:18:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 shutdown_connections 2023-11-08T08:18:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f239c0ced70 0x7f239c1522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2388068320 0x7f238806a7d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06feb0 0x7f239c0992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 --2- 172.21.15.116:0/4187086359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c072420 0x7f239c153480 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:46.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 >> 172.21.15.116:0/4187086359 conn(0x7f239c0b2fc0 msgr2=0x7f239c06dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:46.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 shutdown_connections 2023-11-08T08:18:46.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:46.939+0000 7f23a32f6700 1 -- 172.21.15.116:0/4187086359 wait complete. 2023-11-08T08:18:46.945 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:18:46.951 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-11-08T08:18:46.352511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65796,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":754908924,"statfs":{"total":773094113280,"available":773026738176,"internally_reserved":0,"allocated":2887680,"data_stored":1779623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.873937"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899181+0000","last_change":"2023-11-08T08:18:42.907639+0000","last_active":"2023-11-08T08:18:43.899181+0000","last_peered":"2023-11-08T08:18:43.899181+0000","last_clean":"2023-11-08T08:18:43.899181+0000","last_became_active":"2023-11-08T08:18:42.906449+0000","last_became_peered":"2023-11-08T08:18:42.906449+0000","last_unstale":"2023-11-08T08:18:43.899181+0000","last_undegraded":"2023-11-08T08:18:43.899181+0000","last_fullsized":"2023-11-08T08:18:43.899181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893783+0000","last_change":"2023-11-08T08:18:42.912729+0000","last_active":"2023-11-08T08:18:43.893783+0000","last_peered":"2023-11-08T08:18:43.893783+0000","last_clean":"2023-11-08T08:18:43.893783+0000","last_became_active":"2023-11-08T08:18:42.912482+0000","last_became_peered":"2023-11-08T08:18:42.912482+0000","last_unstale":"2023-11-08T08:18:43.893783+0000","last_undegraded":"2023-11-08T08:18:43.893783+0000","last_fullsized":"2023-11-08T08:18:43.893783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272081+0000","last_change":"2023-11-08T08:18:42.904767+0000","last_active":"2023-11-08T08:18:44.272081+0000","last_peered":"2023-11-08T08:18:44.272081+0000","last_clean":"2023-11-08T08:18:44.272081+0000","last_became_active":"2023-11-08T08:18:42.904512+0000","last_became_peered":"2023-11-08T08:18:42.904512+0000","last_unstale":"2023-11-08T08:18:44.272081+0000","last_undegraded":"2023-11-08T08:18:44.272081+0000","last_fullsized":"2023-11-08T08:18:44.272081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271750+0000","last_change":"2023-11-08T08:18:42.901716+0000","last_active":"2023-11-08T08:18:44.271750+0000","last_peered":"2023-11-08T08:18:44.271750+0000","last_clean":"2023-11-08T08:18:44.271750+0000","last_became_active":"2023-11-08T08:18:42.901477+0000","last_became_peered":"2023-11-08T08:18:42.901477+0000","last_unstale":"2023-11-08T08:18:44.271750+0000","last_undegraded":"2023-11-08T08:18:44.271750+0000","last_fullsized":"2023-11-08T08:18:44.271750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272162+0000","last_change":"2023-11-08T08:18:42.906339+0000","last_active":"2023-11-08T08:18:44.272162+0000","last_peered":"2023-11-08T08:18:44.272162+0000","last_clean":"2023-11-08T08:18:44.272162+0000","last_became_active":"2023-11-08T08:18:42.906082+0000","last_became_peered":"2023-11-08T08:18:42.906082+0000","last_unstale":"2023-11-08T08:18:44.272162+0000","last_undegraded":"2023-11-08T08:18:44.272162+0000","last_fullsized":"2023-11-08T08:18:44.272162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272523+0000","last_change":"2023-11-08T08:18:42.907735+0000","last_active":"2023-11-08T08:18:44.272523+0000","last_peered":"2023-11-08T08:18:44.272523+0000","last_clean":"2023-11-08T08:18:44.272523+0000","last_became_active":"2023-11-08T08:18:42.907414+0000","last_became_peered":"2023-11-08T08:18:42.907414+0000","last_unstale":"2023-11-08T08:18:44.272523+0000","last_undegraded":"2023-11-08T08:18:44.272523+0000","last_fullsized":"2023-11-08T08:18:44.272523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271898+0000","last_change":"2023-11-08T08:18:42.904968+0000","last_active":"2023-11-08T08:18:44.271898+0000","last_peered":"2023-11-08T08:18:44.271898+0000","last_clean":"2023-11-08T08:18:44.271898+0000","last_became_active":"2023-11-08T08:18:42.904586+0000","last_became_peered":"2023-11-08T08:18:42.904586+0000","last_unstale":"2023-11-08T08:18:44.271898+0000","last_undegraded":"2023-11-08T08:18:44.271898+0000","last_fullsized":"2023-11-08T08:18:44.271898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:08.733824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898246+0000","last_change":"2023-11-08T08:18:42.898509+0000","last_active":"2023-11-08T08:18:43.898246+0000","last_peered":"2023-11-08T08:18:43.898246+0000","last_clean":"2023-11-08T08:18:43.898246+0000","last_became_active":"2023-11-08T08:18:42.898230+0000","last_became_peered":"2023-11-08T08:18:42.898230+0000","last_unstale":"2023-11-08T08:18:43.898246+0000","last_undegraded":"2023-11-08T08:18:43.898246+0000","last_fullsized":"2023-11-08T08:18:43.898246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273075+0000","last_change":"2023-11-08T08:18:42.906809+0000","last_active":"2023-11-08T08:18:44.273075+0000","last_peered":"2023-11-08T08:18:44.273075+0000","last_clean":"2023-11-08T08:18:44.273075+0000","last_became_active":"2023-11-08T08:18:42.906517+0000","last_became_peered":"2023-11-08T08:18:42.906517+0000","last_unstale":"2023-11-08T08:18:44.273075+0000","last_undegraded":"2023-11-08T08:18:44.273075+0000","last_fullsized":"2023-11-08T08:18:44.273075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898585+0000","last_change":"2023-11-08T08:18:42.906651+0000","last_active":"2023-11-08T08:18:43.898585+0000","last_peered":"2023-11-08T08:18:43.898585+0000","last_clean":"2023-11-08T08:18:43.898585+0000","last_became_active":"2023-11-08T08:18:42.906379+0000","last_became_peered":"2023-11-08T08:18:42.906379+0000","last_unstale":"2023-11-08T08:18:43.898585+0000","last_undegraded":"2023-11-08T08:18:43.898585+0000","last_fullsized":"2023-11-08T08:18:43.898585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898449+0000","last_change":"2023-11-08T08:18:42.898832+0000","last_active":"2023-11-08T08:18:43.898449+0000","last_peered":"2023-11-08T08:18:43.898449+0000","last_clean":"2023-11-08T08:18:43.898449+0000","last_became_active":"2023-11-08T08:18:42.898581+0000","last_became_peered":"2023-11-08T08:18:42.898581+0000","last_unstale":"2023-11-08T08:18:43.898449+0000","last_undegraded":"2023-11-08T08:18:43.898449+0000","last_fullsized":"2023-11-08T08:18:43.898449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898948+0000","last_change":"2023-11-08T08:18:42.911043+0000","last_active":"2023-11-08T08:18:43.898948+0000","last_peered":"2023-11-08T08:18:43.898948+0000","last_clean":"2023-11-08T08:18:43.898948+0000","last_became_active":"2023-11-08T08:18:42.910756+0000","last_became_peered":"2023-11-08T08:18:42.910756+0000","last_unstale":"2023-11-08T08:18:43.898948+0000","last_undegraded":"2023-11-08T08:18:43.898948+0000","last_fullsized":"2023-11-08T08:18:43.898948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272630+0000","last_change":"2023-11-08T08:18:42.915118+0000","last_active":"2023-11-08T08:18:44.272630+0000","last_peered":"2023-11-08T08:18:44.272630+0000","last_clean":"2023-11-08T08:18:44.272630+0000","last_became_active":"2023-11-08T08:18:42.914854+0000","last_became_peered":"2023-11-08T08:18:42.914854+0000","last_unstale":"2023-11-08T08:18:44.272630+0000","last_undegraded":"2023-11-08T08:18:44.272630+0000","last_fullsized":"2023-11-08T08:18:44.272630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898604+0000","last_change":"2023-11-08T08:18:42.900369+0000","last_active":"2023-11-08T08:18:43.898604+0000","last_peered":"2023-11-08T08:18:43.898604+0000","last_clean":"2023-11-08T08:18:43.898604+0000","last_became_active":"2023-11-08T08:18:42.900121+0000","last_became_peered":"2023-11-08T08:18:42.900121+0000","last_unstale":"2023-11-08T08:18:43.898604+0000","last_undegraded":"2023-11-08T08:18:43.898604+0000","last_fullsized":"2023-11-08T08:18:43.898604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271716+0000","last_change":"2023-11-08T08:18:42.912567+0000","last_active":"2023-11-08T08:18:44.271716+0000","last_peered":"2023-11-08T08:18:44.271716+0000","last_clean":"2023-11-08T08:18:44.271716+0000","last_became_active":"2023-11-08T08:18:42.912299+0000","last_became_peered":"2023-11-08T08:18:42.912299+0000","last_unstale":"2023-11-08T08:18:44.271716+0000","last_undegraded":"2023-11-08T08:18:44.271716+0000","last_fullsized":"2023-11-08T08:18:44.271716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271758+0000","last_change":"2023-11-08T08:18:42.902375+0000","last_active":"2023-11-08T08:18:44.271758+0000","last_peered":"2023-11-08T08:18:44.271758+0000","last_clean":"2023-11-08T08:18:44.271758+0000","last_became_active":"2023-11-08T08:18:42.901640+0000","last_became_peered":"2023-11-08T08:18:42.901640+0000","last_unstale":"2023-11-08T08:18:44.271758+0000","last_undegraded":"2023-11-08T08:18:44.271758+0000","last_fullsized":"2023-11-08T08:18:44.271758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272211+0000","last_change":"2023-11-08T08:18:42.905755+0000","last_active":"2023-11-08T08:18:44.272211+0000","last_peered":"2023-11-08T08:18:44.272211+0000","last_clean":"2023-11-08T08:18:44.272211+0000","last_became_active":"2023-11-08T08:18:42.905498+0000","last_became_peered":"2023-11-08T08:18:42.905498+0000","last_unstale":"2023-11-08T08:18:44.272211+0000","last_undegraded":"2023-11-08T08:18:44.272211+0000","last_fullsized":"2023-11-08T08:18:44.272211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272018+0000","last_change":"2023-11-08T08:18:42.900992+0000","last_active":"2023-11-08T08:18:44.272018+0000","last_peered":"2023-11-08T08:18:44.272018+0000","last_clean":"2023-11-08T08:18:44.272018+0000","last_became_active":"2023-11-08T08:18:42.900747+0000","last_became_peered":"2023-11-08T08:18:42.900747+0000","last_unstale":"2023-11-08T08:18:44.272018+0000","last_undegraded":"2023-11-08T08:18:44.272018+0000","last_fullsized":"2023-11-08T08:18:44.272018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899009+0000","last_change":"2023-11-08T08:18:42.910975+0000","last_active":"2023-11-08T08:18:43.899009+0000","last_peered":"2023-11-08T08:18:43.899009+0000","last_clean":"2023-11-08T08:18:43.899009+0000","last_became_active":"2023-11-08T08:18:42.910701+0000","last_became_peered":"2023-11-08T08:18:42.910701+0000","last_unstale":"2023-11-08T08:18:43.899009+0000","last_undegraded":"2023-11-08T08:18:43.899009+0000","last_fullsized":"2023-11-08T08:18:43.899009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898283+0000","last_change":"2023-11-08T08:18:42.912881+0000","last_active":"2023-11-08T08:18:43.898283+0000","last_peered":"2023-11-08T08:18:43.898283+0000","last_clean":"2023-11-08T08:18:43.898283+0000","last_became_active":"2023-11-08T08:18:42.912660+0000","last_became_peered":"2023-11-08T08:18:42.912660+0000","last_unstale":"2023-11-08T08:18:43.898283+0000","last_undegraded":"2023-11-08T08:18:43.898283+0000","last_fullsized":"2023-11-08T08:18:43.898283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898341+0000","last_change":"2023-11-08T08:18:42.901149+0000","last_active":"2023-11-08T08:18:43.898341+0000","last_peered":"2023-11-08T08:18:43.898341+0000","last_clean":"2023-11-08T08:18:43.898341+0000","last_became_active":"2023-11-08T08:18:42.900728+0000","last_became_peered":"2023-11-08T08:18:42.900728+0000","last_unstale":"2023-11-08T08:18:43.898341+0000","last_undegraded":"2023-11-08T08:18:43.898341+0000","last_fullsized":"2023-11-08T08:18:43.898341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272020+0000","last_change":"2023-11-08T08:18:42.911764+0000","last_active":"2023-11-08T08:18:44.272020+0000","last_peered":"2023-11-08T08:18:44.272020+0000","last_clean":"2023-11-08T08:18:44.272020+0000","last_became_active":"2023-11-08T08:18:42.911477+0000","last_became_peered":"2023-11-08T08:18:42.911477+0000","last_unstale":"2023-11-08T08:18:44.272020+0000","last_undegraded":"2023-11-08T08:18:44.272020+0000","last_fullsized":"2023-11-08T08:18:44.272020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272570+0000","last_change":"2023-11-08T08:18:42.909892+0000","last_active":"2023-11-08T08:18:44.272570+0000","last_peered":"2023-11-08T08:18:44.272570+0000","last_clean":"2023-11-08T08:18:44.272570+0000","last_became_active":"2023-11-08T08:18:42.909646+0000","last_became_peered":"2023-11-08T08:18:42.909646+0000","last_unstale":"2023-11-08T08:18:44.272570+0000","last_undegraded":"2023-11-08T08:18:44.272570+0000","last_fullsized":"2023-11-08T08:18:44.272570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898218+0000","last_change":"2023-11-08T08:18:42.897278+0000","last_active":"2023-11-08T08:18:43.898218+0000","last_peered":"2023-11-08T08:18:43.898218+0000","last_clean":"2023-11-08T08:18:43.898218+0000","last_became_active":"2023-11-08T08:18:42.896923+0000","last_became_peered":"2023-11-08T08:18:42.896923+0000","last_unstale":"2023-11-08T08:18:43.898218+0000","last_undegraded":"2023-11-08T08:18:43.898218+0000","last_fullsized":"2023-11-08T08:18:43.898218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273144+0000","last_change":"2023-11-08T08:18:42.909514+0000","last_active":"2023-11-08T08:18:44.273144+0000","last_peered":"2023-11-08T08:18:44.273144+0000","last_clean":"2023-11-08T08:18:44.273144+0000","last_became_active":"2023-11-08T08:18:42.909246+0000","last_became_peered":"2023-11-08T08:18:42.909246+0000","last_unstale":"2023-11-08T08:18:44.273144+0000","last_undegraded":"2023-11-08T08:18:44.273144+0000","last_fullsized":"2023-11-08T08:18:44.273144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272924+0000","last_change":"2023-11-08T08:18:42.912860+0000","last_active":"2023-11-08T08:18:44.272924+0000","last_peered":"2023-11-08T08:18:44.272924+0000","last_clean":"2023-11-08T08:18:44.272924+0000","last_became_active":"2023-11-08T08:18:42.912613+0000","last_became_peered":"2023-11-08T08:18:42.912613+0000","last_unstale":"2023-11-08T08:18:44.272924+0000","last_undegraded":"2023-11-08T08:18:44.272924+0000","last_fullsized":"2023-11-08T08:18:44.272924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272694+0000","last_change":"2023-11-08T08:18:42.898517+0000","last_active":"2023-11-08T08:18:44.272694+0000","last_peered":"2023-11-08T08:18:44.272694+0000","last_clean":"2023-11-08T08:18:44.272694+0000","last_became_active":"2023-11-08T08:18:42.898253+0000","last_became_peered":"2023-11-08T08:18:42.898253+0000","last_unstale":"2023-11-08T08:18:44.272694+0000","last_undegraded":"2023-11-08T08:18:44.272694+0000","last_fullsized":"2023-11-08T08:18:44.272694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:56:57.876135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898142+0000","last_change":"2023-11-08T08:18:42.501016+0000","last_active":"2023-11-08T08:18:43.898142+0000","last_peered":"2023-11-08T08:18:43.898142+0000","last_clean":"2023-11-08T08:18:43.898142+0000","last_became_active":"2023-11-08T08:18:42.499597+0000","last_became_peered":"2023-11-08T08:18:42.499597+0000","last_unstale":"2023-11-08T08:18:43.898142+0000","last_undegraded":"2023-11-08T08:18:43.898142+0000","last_fullsized":"2023-11-08T08:18:43.898142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272484+0000","last_change":"2023-11-08T08:18:42.907418+0000","last_active":"2023-11-08T08:18:44.272484+0000","last_peered":"2023-11-08T08:18:44.272484+0000","last_clean":"2023-11-08T08:18:44.272484+0000","last_became_active":"2023-11-08T08:18:42.907045+0000","last_became_peered":"2023-11-08T08:18:42.907045+0000","last_unstale":"2023-11-08T08:18:44.272484+0000","last_undegraded":"2023-11-08T08:18:44.272484+0000","last_fullsized":"2023-11-08T08:18:44.272484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272507+0000","last_change":"2023-11-08T08:18:42.497426+0000","last_active":"2023-11-08T08:18:44.272507+0000","last_peered":"2023-11-08T08:18:44.272507+0000","last_clean":"2023-11-08T08:18:44.272507+0000","last_became_active":"2023-11-08T08:18:42.497130+0000","last_became_peered":"2023-11-08T08:18:42.497130+0000","last_unstale":"2023-11-08T08:18:44.272507+0000","last_undegraded":"2023-11-08T08:18:44.272507+0000","last_fullsized":"2023-11-08T08:18:44.272507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272906+0000","last_change":"2023-11-08T08:18:42.905727+0000","last_active":"2023-11-08T08:18:44.272906+0000","last_peered":"2023-11-08T08:18:44.272906+0000","last_clean":"2023-11-08T08:18:44.272906+0000","last_became_active":"2023-11-08T08:18:42.905467+0000","last_became_peered":"2023-11-08T08:18:42.905467+0000","last_unstale":"2023-11-08T08:18:44.272906+0000","last_undegraded":"2023-11-08T08:18:44.272906+0000","last_fullsized":"2023-11-08T08:18:44.272906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898639+0000","last_change":"2023-11-08T08:18:42.502963+0000","last_active":"2023-11-08T08:18:43.898639+0000","last_peered":"2023-11-08T08:18:43.898639+0000","last_clean":"2023-11-08T08:18:43.898639+0000","last_became_active":"2023-11-08T08:18:42.502702+0000","last_became_peered":"2023-11-08T08:18:42.502702+0000","last_unstale":"2023-11-08T08:18:43.898639+0000","last_undegraded":"2023-11-08T08:18:43.898639+0000","last_fullsized":"2023-11-08T08:18:43.898639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272590+0000","last_change":"2023-11-08T08:18:42.904909+0000","last_active":"2023-11-08T08:18:44.272590+0000","last_peered":"2023-11-08T08:18:44.272590+0000","last_clean":"2023-11-08T08:18:44.272590+0000","last_became_active":"2023-11-08T08:18:42.903891+0000","last_became_peered":"2023-11-08T08:18:42.903891+0000","last_unstale":"2023-11-08T08:18:44.272590+0000","last_undegraded":"2023-11-08T08:18:44.272590+0000","last_fullsized":"2023-11-08T08:18:44.272590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:21.735174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899300+0000","last_change":"2023-11-08T08:18:42.498275+0000","last_active":"2023-11-08T08:18:43.899300+0000","last_peered":"2023-11-08T08:18:43.899300+0000","last_clean":"2023-11-08T08:18:43.899300+0000","last_became_active":"2023-11-08T08:18:42.497975+0000","last_became_peered":"2023-11-08T08:18:42.497975+0000","last_unstale":"2023-11-08T08:18:43.899300+0000","last_undegraded":"2023-11-08T08:18:43.899300+0000","last_fullsized":"2023-11-08T08:18:43.899300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271740+0000","last_change":"2023-11-08T08:18:42.900693+0000","last_active":"2023-11-08T08:18:44.271740+0000","last_peered":"2023-11-08T08:18:44.271740+0000","last_clean":"2023-11-08T08:18:44.271740+0000","last_became_active":"2023-11-08T08:18:42.900407+0000","last_became_peered":"2023-11-08T08:18:42.900407+0000","last_unstale":"2023-11-08T08:18:44.271740+0000","last_undegraded":"2023-11-08T08:18:44.271740+0000","last_fullsized":"2023-11-08T08:18:44.271740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894013+0000","last_change":"2023-11-08T08:18:42.502815+0000","last_active":"2023-11-08T08:18:43.894013+0000","last_peered":"2023-11-08T08:18:43.894013+0000","last_clean":"2023-11-08T08:18:43.894013+0000","last_became_active":"2023-11-08T08:18:42.502329+0000","last_became_peered":"2023-11-08T08:18:42.502329+0000","last_unstale":"2023-11-08T08:18:43.894013+0000","last_undegraded":"2023-11-08T08:18:43.894013+0000","last_fullsized":"2023-11-08T08:18:43.894013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272456+0000","last_change":"2023-11-08T08:18:42.909334+0000","last_active":"2023-11-08T08:18:44.272456+0000","last_peered":"2023-11-08T08:18:44.272456+0000","last_clean":"2023-11-08T08:18:44.272456+0000","last_became_active":"2023-11-08T08:18:42.907416+0000","last_became_peered":"2023-11-08T08:18:42.907416+0000","last_unstale":"2023-11-08T08:18:44.272456+0000","last_undegraded":"2023-11-08T08:18:44.272456+0000","last_fullsized":"2023-11-08T08:18:44.272456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272167+0000","last_change":"2023-11-08T08:18:42.504744+0000","last_active":"2023-11-08T08:18:44.272167+0000","last_peered":"2023-11-08T08:18:44.272167+0000","last_clean":"2023-11-08T08:18:44.272167+0000","last_became_active":"2023-11-08T08:18:42.504438+0000","last_became_peered":"2023-11-08T08:18:42.504438+0000","last_unstale":"2023-11-08T08:18:44.272167+0000","last_undegraded":"2023-11-08T08:18:44.272167+0000","last_fullsized":"2023-11-08T08:18:44.272167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272646+0000","last_change":"2023-11-08T08:18:42.910922+0000","last_active":"2023-11-08T08:18:44.272646+0000","last_peered":"2023-11-08T08:18:44.272646+0000","last_clean":"2023-11-08T08:18:44.272646+0000","last_became_active":"2023-11-08T08:18:42.910578+0000","last_became_peered":"2023-11-08T08:18:42.910578+0000","last_unstale":"2023-11-08T08:18:44.272646+0000","last_undegraded":"2023-11-08T08:18:44.272646+0000","last_fullsized":"2023-11-08T08:18:44.272646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272486+0000","last_change":"2023-11-08T08:18:42.497349+0000","last_active":"2023-11-08T08:18:44.272486+0000","last_peered":"2023-11-08T08:18:44.272486+0000","last_clean":"2023-11-08T08:18:44.272486+0000","last_became_active":"2023-11-08T08:18:42.497038+0000","last_became_peered":"2023-11-08T08:18:42.497038+0000","last_unstale":"2023-11-08T08:18:44.272486+0000","last_undegraded":"2023-11-08T08:18:44.272486+0000","last_fullsized":"2023-11-08T08:18:44.272486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:12.090061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894075+0000","last_change":"2023-11-08T08:18:42.905780+0000","last_active":"2023-11-08T08:18:43.894075+0000","last_peered":"2023-11-08T08:18:43.894075+0000","last_clean":"2023-11-08T08:18:43.894075+0000","last_became_active":"2023-11-08T08:18:42.905487+0000","last_became_peered":"2023-11-08T08:18:42.905487+0000","last_unstale":"2023-11-08T08:18:43.894075+0000","last_undegraded":"2023-11-08T08:18:43.894075+0000","last_fullsized":"2023-11-08T08:18:43.894075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272417+0000","last_change":"2023-11-08T08:18:42.504787+0000","last_active":"2023-11-08T08:18:44.272417+0000","last_peered":"2023-11-08T08:18:44.272417+0000","last_clean":"2023-11-08T08:18:44.272417+0000","last_became_active":"2023-11-08T08:18:42.504477+0000","last_became_peered":"2023-11-08T08:18:42.504477+0000","last_unstale":"2023-11-08T08:18:44.272417+0000","last_undegraded":"2023-11-08T08:18:44.272417+0000","last_fullsized":"2023-11-08T08:18:44.272417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:38.511975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272076+0000","last_change":"2023-11-08T08:18:42.913147+0000","last_active":"2023-11-08T08:18:44.272076+0000","last_peered":"2023-11-08T08:18:44.272076+0000","last_clean":"2023-11-08T08:18:44.272076+0000","last_became_active":"2023-11-08T08:18:42.912855+0000","last_became_peered":"2023-11-08T08:18:42.912855+0000","last_unstale":"2023-11-08T08:18:44.272076+0000","last_undegraded":"2023-11-08T08:18:44.272076+0000","last_fullsized":"2023-11-08T08:18:44.272076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272506+0000","last_change":"2023-11-08T08:18:42.504414+0000","last_active":"2023-11-08T08:18:44.272506+0000","last_peered":"2023-11-08T08:18:44.272506+0000","last_clean":"2023-11-08T08:18:44.272506+0000","last_became_active":"2023-11-08T08:18:42.504141+0000","last_became_peered":"2023-11-08T08:18:42.504141+0000","last_unstale":"2023-11-08T08:18:44.272506+0000","last_undegraded":"2023-11-08T08:18:44.272506+0000","last_fullsized":"2023-11-08T08:18:44.272506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272647+0000","last_change":"2023-11-08T08:18:42.908975+0000","last_active":"2023-11-08T08:18:44.272647+0000","last_peered":"2023-11-08T08:18:44.272647+0000","last_clean":"2023-11-08T08:18:44.272647+0000","last_became_active":"2023-11-08T08:18:42.908137+0000","last_became_peered":"2023-11-08T08:18:42.908137+0000","last_unstale":"2023-11-08T08:18:44.272647+0000","last_undegraded":"2023-11-08T08:18:44.272647+0000","last_fullsized":"2023-11-08T08:18:44.272647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:35:59.915321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.495958+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.495711+0000","last_became_peered":"2023-11-08T08:18:42.495711+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:04.545064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899087+0000","last_change":"2023-11-08T08:18:42.907135+0000","last_active":"2023-11-08T08:18:43.899087+0000","last_peered":"2023-11-08T08:18:43.899087+0000","last_clean":"2023-11-08T08:18:43.899087+0000","last_became_active":"2023-11-08T08:18:42.906893+0000","last_became_peered":"2023-11-08T08:18:42.906893+0000","last_unstale":"2023-11-08T08:18:43.899087+0000","last_undegraded":"2023-11-08T08:18:43.899087+0000","last_fullsized":"2023-11-08T08:18:43.899087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898115+0000","last_change":"2023-11-08T08:18:42.505141+0000","last_active":"2023-11-08T08:18:43.898115+0000","last_peered":"2023-11-08T08:18:43.898115+0000","last_clean":"2023-11-08T08:18:43.898115+0000","last_became_active":"2023-11-08T08:18:42.500393+0000","last_became_peered":"2023-11-08T08:18:42.500393+0000","last_unstale":"2023-11-08T08:18:43.898115+0000","last_undegraded":"2023-11-08T08:18:43.898115+0000","last_fullsized":"2023-11-08T08:18:43.898115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899261+0000","last_change":"2023-11-08T08:18:42.908933+0000","last_active":"2023-11-08T08:18:43.899261+0000","last_peered":"2023-11-08T08:18:43.899261+0000","last_clean":"2023-11-08T08:18:43.899261+0000","last_became_active":"2023-11-08T08:18:42.908643+0000","last_became_peered":"2023-11-08T08:18:42.908643+0000","last_unstale":"2023-11-08T08:18:43.899261+0000","last_undegraded":"2023-11-08T08:18:43.899261+0000","last_fullsized":"2023-11-08T08:18:43.899261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272143+0000","last_change":"2023-11-08T08:18:42.506315+0000","last_active":"2023-11-08T08:18:44.272143+0000","last_peered":"2023-11-08T08:18:44.272143+0000","last_clean":"2023-11-08T08:18:44.272143+0000","last_became_active":"2023-11-08T08:18:42.506044+0000","last_became_peered":"2023-11-08T08:18:42.506044+0000","last_unstale":"2023-11-08T08:18:44.272143+0000","last_undegraded":"2023-11-08T08:18:44.272143+0000","last_fullsized":"2023-11-08T08:18:44.272143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271908+0000","last_change":"2023-11-08T08:18:42.908490+0000","last_active":"2023-11-08T08:18:44.271908+0000","last_peered":"2023-11-08T08:18:44.271908+0000","last_clean":"2023-11-08T08:18:44.271908+0000","last_became_active":"2023-11-08T08:18:42.908302+0000","last_became_peered":"2023-11-08T08:18:42.908302+0000","last_unstale":"2023-11-08T08:18:44.271908+0000","last_undegraded":"2023-11-08T08:18:44.271908+0000","last_fullsized":"2023-11-08T08:18:44.271908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893984+0000","last_change":"2023-11-08T08:18:42.504388+0000","last_active":"2023-11-08T08:18:43.893984+0000","last_peered":"2023-11-08T08:18:43.893984+0000","last_clean":"2023-11-08T08:18:43.893984+0000","last_became_active":"2023-11-08T08:18:42.504085+0000","last_became_peered":"2023-11-08T08:18:42.504085+0000","last_unstale":"2023-11-08T08:18:43.893984+0000","last_undegraded":"2023-11-08T08:18:43.893984+0000","last_fullsized":"2023-11-08T08:18:43.893984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898126+0000","last_change":"2023-11-08T08:18:42.901943+0000","last_active":"2023-11-08T08:18:43.898126+0000","last_peered":"2023-11-08T08:18:43.898126+0000","last_clean":"2023-11-08T08:18:43.898126+0000","last_became_active":"2023-11-08T08:18:42.901687+0000","last_became_peered":"2023-11-08T08:18:42.901687+0000","last_unstale":"2023-11-08T08:18:43.898126+0000","last_undegraded":"2023-11-08T08:18:43.898126+0000","last_fullsized":"2023-11-08T08:18:43.898126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272582+0000","last_change":"2023-11-08T08:18:42.502289+0000","last_active":"2023-11-08T08:18:44.272582+0000","last_peered":"2023-11-08T08:18:44.272582+0000","last_clean":"2023-11-08T08:18:44.272582+0000","last_became_active":"2023-11-08T08:18:42.499404+0000","last_became_peered":"2023-11-08T08:18:42.499404+0000","last_unstale":"2023-11-08T08:18:44.272582+0000","last_undegraded":"2023-11-08T08:18:44.272582+0000","last_fullsized":"2023-11-08T08:18:44.272582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:43.337457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893816+0000","last_change":"2023-11-08T08:18:42.905468+0000","last_active":"2023-11-08T08:18:43.893816+0000","last_peered":"2023-11-08T08:18:43.893816+0000","last_clean":"2023-11-08T08:18:43.893816+0000","last_became_active":"2023-11-08T08:18:42.905203+0000","last_became_peered":"2023-11-08T08:18:42.905203+0000","last_unstale":"2023-11-08T08:18:43.893816+0000","last_undegraded":"2023-11-08T08:18:43.893816+0000","last_fullsized":"2023-11-08T08:18:43.893816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.503204+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.501193+0000","last_became_peered":"2023-11-08T08:18:42.501193+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899051+0000","last_change":"2023-11-08T08:18:42.911979+0000","last_active":"2023-11-08T08:18:43.899051+0000","last_peered":"2023-11-08T08:18:43.899051+0000","last_clean":"2023-11-08T08:18:43.899051+0000","last_became_active":"2023-11-08T08:18:42.911070+0000","last_became_peered":"2023-11-08T08:18:42.911070+0000","last_unstale":"2023-11-08T08:18:43.899051+0000","last_undegraded":"2023-11-08T08:18:43.899051+0000","last_fullsized":"2023-11-08T08:18:43.899051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272150+0000","last_change":"2023-11-08T08:18:42.503625+0000","last_active":"2023-11-08T08:18:44.272150+0000","last_peered":"2023-11-08T08:18:44.272150+0000","last_clean":"2023-11-08T08:18:44.272150+0000","last_became_active":"2023-11-08T08:18:42.503351+0000","last_became_peered":"2023-11-08T08:18:42.503351+0000","last_unstale":"2023-11-08T08:18:44.272150+0000","last_undegraded":"2023-11-08T08:18:44.272150+0000","last_fullsized":"2023-11-08T08:18:44.272150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271896+0000","last_change":"2023-11-08T08:18:42.910859+0000","last_active":"2023-11-08T08:18:44.271896+0000","last_peered":"2023-11-08T08:18:44.271896+0000","last_clean":"2023-11-08T08:18:44.271896+0000","last_became_active":"2023-11-08T08:18:42.910598+0000","last_became_peered":"2023-11-08T08:18:42.910598+0000","last_unstale":"2023-11-08T08:18:44.271896+0000","last_undegraded":"2023-11-08T08:18:44.271896+0000","last_fullsized":"2023-11-08T08:18:44.271896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272298+0000","last_change":"2023-11-08T08:18:42.499085+0000","last_active":"2023-11-08T08:18:44.272298+0000","last_peered":"2023-11-08T08:18:44.272298+0000","last_clean":"2023-11-08T08:18:44.272298+0000","last_became_active":"2023-11-08T08:18:42.498750+0000","last_became_peered":"2023-11-08T08:18:42.498750+0000","last_unstale":"2023-11-08T08:18:44.272298+0000","last_undegraded":"2023-11-08T08:18:44.272298+0000","last_fullsized":"2023-11-08T08:18:44.272298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272549+0000","last_change":"2023-11-08T08:18:42.903923+0000","last_active":"2023-11-08T08:18:44.272549+0000","last_peered":"2023-11-08T08:18:44.272549+0000","last_clean":"2023-11-08T08:18:44.272549+0000","last_became_active":"2023-11-08T08:18:42.903518+0000","last_became_peered":"2023-11-08T08:18:42.903518+0000","last_unstale":"2023-11-08T08:18:44.272549+0000","last_undegraded":"2023-11-08T08:18:44.272549+0000","last_fullsized":"2023-11-08T08:18:44.272549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:51:16.185661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272139+0000","last_change":"2023-11-08T08:18:42.496450+0000","last_active":"2023-11-08T08:18:44.272139+0000","last_peered":"2023-11-08T08:18:44.272139+0000","last_clean":"2023-11-08T08:18:44.272139+0000","last_became_active":"2023-11-08T08:18:42.496197+0000","last_became_peered":"2023-11-08T08:18:42.496197+0000","last_unstale":"2023-11-08T08:18:44.272139+0000","last_undegraded":"2023-11-08T08:18:44.272139+0000","last_fullsized":"2023-11-08T08:18:44.272139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893897+0000","last_change":"2023-11-08T08:18:42.906952+0000","last_active":"2023-11-08T08:18:43.893897+0000","last_peered":"2023-11-08T08:18:43.893897+0000","last_clean":"2023-11-08T08:18:43.893897+0000","last_became_active":"2023-11-08T08:18:42.906699+0000","last_became_peered":"2023-11-08T08:18:42.906699+0000","last_unstale":"2023-11-08T08:18:43.893897+0000","last_undegraded":"2023-11-08T08:18:43.893897+0000","last_fullsized":"2023-11-08T08:18:43.893897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271876+0000","last_change":"2023-11-08T08:18:42.496216+0000","last_active":"2023-11-08T08:18:44.271876+0000","last_peered":"2023-11-08T08:18:44.271876+0000","last_clean":"2023-11-08T08:18:44.271876+0000","last_became_active":"2023-11-08T08:18:42.495940+0000","last_became_peered":"2023-11-08T08:18:42.495940+0000","last_unstale":"2023-11-08T08:18:44.271876+0000","last_undegraded":"2023-11-08T08:18:44.271876+0000","last_fullsized":"2023-11-08T08:18:44.271876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272347+0000","last_change":"2023-11-08T08:18:42.916881+0000","last_active":"2023-11-08T08:18:44.272347+0000","last_peered":"2023-11-08T08:18:44.272347+0000","last_clean":"2023-11-08T08:18:44.272347+0000","last_became_active":"2023-11-08T08:18:42.916593+0000","last_became_peered":"2023-11-08T08:18:42.916593+0000","last_unstale":"2023-11-08T08:18:44.272347+0000","last_undegraded":"2023-11-08T08:18:44.272347+0000","last_fullsized":"2023-11-08T08:18:44.272347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:09.598915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273120+0000","last_change":"2023-11-08T08:18:42.502915+0000","last_active":"2023-11-08T08:18:44.273120+0000","last_peered":"2023-11-08T08:18:44.273120+0000","last_clean":"2023-11-08T08:18:44.273120+0000","last_became_active":"2023-11-08T08:18:42.502675+0000","last_became_peered":"2023-11-08T08:18:42.502675+0000","last_unstale":"2023-11-08T08:18:44.273120+0000","last_undegraded":"2023-11-08T08:18:44.273120+0000","last_fullsized":"2023-11-08T08:18:44.273120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:08.593928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.891523+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:18:42.891523+0000","last_peered":"2023-11-08T08:18:42.891523+0000","last_clean":"2023-11-08T08:18:42.891523+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:18:42.891523+0000","last_undegraded":"2023-11-08T08:18:42.891523+0000","last_fullsized":"2023-11-08T08:18:42.891523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271984+0000","last_change":"2023-11-08T08:18:42.908748+0000","last_active":"2023-11-08T08:18:44.271984+0000","last_peered":"2023-11-08T08:18:44.271984+0000","last_clean":"2023-11-08T08:18:44.271984+0000","last_became_active":"2023-11-08T08:18:42.908419+0000","last_became_peered":"2023-11-08T08:18:42.908419+0000","last_unstale":"2023-11-08T08:18:44.271984+0000","last_undegraded":"2023-11-08T08:18:44.271984+0000","last_fullsized":"2023-11-08T08:18:44.271984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894809+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:18:42.894809+0000","last_peered":"2023-11-08T08:18:42.894809+0000","last_clean":"2023-11-08T08:18:42.894809+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:18:42.894809+0000","last_undegraded":"2023-11-08T08:18:42.894809+0000","last_fullsized":"2023-11-08T08:18:42.894809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272802+0000","last_change":"2023-11-08T08:18:42.496201+0000","last_active":"2023-11-08T08:18:44.272802+0000","last_peered":"2023-11-08T08:18:44.272802+0000","last_clean":"2023-11-08T08:18:44.272802+0000","last_became_active":"2023-11-08T08:18:42.495914+0000","last_became_peered":"2023-11-08T08:18:42.495914+0000","last_unstale":"2023-11-08T08:18:44.272802+0000","last_undegraded":"2023-11-08T08:18:44.272802+0000","last_fullsized":"2023-11-08T08:18:44.272802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893727+0000","last_change":"2023-11-08T08:18:42.907389+0000","last_active":"2023-11-08T08:18:43.893727+0000","last_peered":"2023-11-08T08:18:43.893727+0000","last_clean":"2023-11-08T08:18:43.893727+0000","last_became_active":"2023-11-08T08:18:42.906767+0000","last_became_peered":"2023-11-08T08:18:42.906767+0000","last_unstale":"2023-11-08T08:18:43.893727+0000","last_undegraded":"2023-11-08T08:18:43.893727+0000","last_fullsized":"2023-11-08T08:18:43.893727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.890889+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:18:42.890889+0000","last_peered":"2023-11-08T08:18:42.890889+0000","last_clean":"2023-11-08T08:18:42.890889+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:18:42.890889+0000","last_undegraded":"2023-11-08T08:18:42.890889+0000","last_fullsized":"2023-11-08T08:18:42.890889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.895954+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:18:42.895954+0000","last_peered":"2023-11-08T08:18:42.895954+0000","last_clean":"2023-11-08T08:18:42.895954+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:18:42.895954+0000","last_undegraded":"2023-11-08T08:18:42.895954+0000","last_fullsized":"2023-11-08T08:18:42.895954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272359+0000","last_change":"2023-11-08T08:18:42.505906+0000","last_active":"2023-11-08T08:18:44.272359+0000","last_peered":"2023-11-08T08:18:44.272359+0000","last_clean":"2023-11-08T08:18:44.272359+0000","last_became_active":"2023-11-08T08:18:42.505688+0000","last_became_peered":"2023-11-08T08:18:42.505688+0000","last_unstale":"2023-11-08T08:18:44.272359+0000","last_undegraded":"2023-11-08T08:18:44.272359+0000","last_fullsized":"2023-11-08T08:18:44.272359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899202+0000","last_change":"2023-11-08T08:18:42.907668+0000","last_active":"2023-11-08T08:18:43.899202+0000","last_peered":"2023-11-08T08:18:43.899202+0000","last_clean":"2023-11-08T08:18:43.899202+0000","last_became_active":"2023-11-08T08:18:42.907352+0000","last_became_peered":"2023-11-08T08:18:42.907352+0000","last_unstale":"2023-11-08T08:18:43.899202+0000","last_undegraded":"2023-11-08T08:18:43.899202+0000","last_fullsized":"2023-11-08T08:18:43.899202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894118+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:18:42.894118+0000","last_peered":"2023-11-08T08:18:42.894118+0000","last_clean":"2023-11-08T08:18:42.894118+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:18:42.894118+0000","last_undegraded":"2023-11-08T08:18:42.894118+0000","last_fullsized":"2023-11-08T08:18:42.894118+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898738+0000","last_change":"2023-11-08T08:18:42.491954+0000","last_active":"2023-11-08T08:18:43.898738+0000","last_peered":"2023-11-08T08:18:43.898738+0000","last_clean":"2023-11-08T08:18:43.898738+0000","last_became_active":"2023-11-08T08:18:42.491305+0000","last_became_peered":"2023-11-08T08:18:42.491305+0000","last_unstale":"2023-11-08T08:18:43.898738+0000","last_undegraded":"2023-11-08T08:18:43.898738+0000","last_fullsized":"2023-11-08T08:18:43.898738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272932+0000","last_change":"2023-11-08T08:18:42.910300+0000","last_active":"2023-11-08T08:18:44.272932+0000","last_peered":"2023-11-08T08:18:44.272932+0000","last_clean":"2023-11-08T08:18:44.272932+0000","last_became_active":"2023-11-08T08:18:42.910084+0000","last_became_peered":"2023-11-08T08:18:42.910084+0000","last_unstale":"2023-11-08T08:18:44.272932+0000","last_undegraded":"2023-11-08T08:18:44.272932+0000","last_fullsized":"2023-11-08T08:18:44.272932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.889516+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:18:42.889516+0000","last_peered":"2023-11-08T08:18:42.889516+0000","last_clean":"2023-11-08T08:18:42.889516+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:18:42.889516+0000","last_undegraded":"2023-11-08T08:18:42.889516+0000","last_fullsized":"2023-11-08T08:18:42.889516+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272717+0000","last_change":"2023-11-08T08:18:42.493859+0000","last_active":"2023-11-08T08:18:44.272717+0000","last_peered":"2023-11-08T08:18:44.272717+0000","last_clean":"2023-11-08T08:18:44.272717+0000","last_became_active":"2023-11-08T08:18:42.492594+0000","last_became_peered":"2023-11-08T08:18:42.492594+0000","last_unstale":"2023-11-08T08:18:44.272717+0000","last_undegraded":"2023-11-08T08:18:44.272717+0000","last_fullsized":"2023-11-08T08:18:44.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272644+0000","last_change":"2023-11-08T08:18:42.903762+0000","last_active":"2023-11-08T08:18:44.272644+0000","last_peered":"2023-11-08T08:18:44.272644+0000","last_clean":"2023-11-08T08:18:44.272644+0000","last_became_active":"2023-11-08T08:18:42.903182+0000","last_became_peered":"2023-11-08T08:18:42.903182+0000","last_unstale":"2023-11-08T08:18:44.272644+0000","last_undegraded":"2023-11-08T08:18:44.272644+0000","last_fullsized":"2023-11-08T08:18:44.272644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:23:54.970595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896906+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:18:42.896906+0000","last_peered":"2023-11-08T08:18:42.896906+0000","last_clean":"2023-11-08T08:18:42.896906+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:18:42.896906+0000","last_undegraded":"2023-11-08T08:18:42.896906+0000","last_fullsized":"2023-11-08T08:18:42.896906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272296+0000","last_change":"2023-11-08T08:18:42.506245+0000","last_active":"2023-11-08T08:18:44.272296+0000","last_peered":"2023-11-08T08:18:44.272296+0000","last_clean":"2023-11-08T08:18:44.272296+0000","last_became_active":"2023-11-08T08:18:42.505955+0000","last_became_peered":"2023-11-08T08:18:42.505955+0000","last_unstale":"2023-11-08T08:18:44.272296+0000","last_undegraded":"2023-11-08T08:18:44.272296+0000","last_fullsized":"2023-11-08T08:18:44.272296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272936+0000","last_change":"2023-11-08T08:18:42.905213+0000","last_active":"2023-11-08T08:18:44.272936+0000","last_peered":"2023-11-08T08:18:44.272936+0000","last_clean":"2023-11-08T08:18:44.272936+0000","last_became_active":"2023-11-08T08:18:42.904949+0000","last_became_peered":"2023-11-08T08:18:42.904949+0000","last_unstale":"2023-11-08T08:18:44.272936+0000","last_undegraded":"2023-11-08T08:18:44.272936+0000","last_fullsized":"2023-11-08T08:18:44.272936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896829+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:18:42.896829+0000","last_peered":"2023-11-08T08:18:42.896829+0000","last_clean":"2023-11-08T08:18:42.896829+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:18:42.896829+0000","last_undegraded":"2023-11-08T08:18:42.896829+0000","last_fullsized":"2023-11-08T08:18:42.896829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272802+0000","last_change":"2023-11-08T08:18:42.502871+0000","last_active":"2023-11-08T08:18:44.272802+0000","last_peered":"2023-11-08T08:18:44.272802+0000","last_clean":"2023-11-08T08:18:44.272802+0000","last_became_active":"2023-11-08T08:18:42.502339+0000","last_became_peered":"2023-11-08T08:18:42.502339+0000","last_unstale":"2023-11-08T08:18:44.272802+0000","last_undegraded":"2023-11-08T08:18:44.272802+0000","last_fullsized":"2023-11-08T08:18:44.272802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:11:07.923861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272804+0000","last_change":"2023-11-08T08:18:42.910283+0000","last_active":"2023-11-08T08:18:44.272804+0000","last_peered":"2023-11-08T08:18:44.272804+0000","last_clean":"2023-11-08T08:18:44.272804+0000","last_became_active":"2023-11-08T08:18:42.909986+0000","last_became_peered":"2023-11-08T08:18:42.909986+0000","last_unstale":"2023-11-08T08:18:44.272804+0000","last_undegraded":"2023-11-08T08:18:44.272804+0000","last_fullsized":"2023-11-08T08:18:44.272804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.892038+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:18:42.892038+0000","last_peered":"2023-11-08T08:18:42.892038+0000","last_clean":"2023-11-08T08:18:42.892038+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:18:42.892038+0000","last_undegraded":"2023-11-08T08:18:42.892038+0000","last_fullsized":"2023-11-08T08:18:42.892038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893889+0000","last_change":"2023-11-08T08:18:42.503455+0000","last_active":"2023-11-08T08:18:43.893889+0000","last_peered":"2023-11-08T08:18:43.893889+0000","last_clean":"2023-11-08T08:18:43.893889+0000","last_became_active":"2023-11-08T08:18:42.503187+0000","last_became_peered":"2023-11-08T08:18:42.503187+0000","last_unstale":"2023-11-08T08:18:43.893889+0000","last_undegraded":"2023-11-08T08:18:43.893889+0000","last_fullsized":"2023-11-08T08:18:43.893889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272626+0000","last_change":"2023-11-08T08:18:42.911640+0000","last_active":"2023-11-08T08:18:44.272626+0000","last_peered":"2023-11-08T08:18:44.272626+0000","last_clean":"2023-11-08T08:18:44.272626+0000","last_became_active":"2023-11-08T08:18:42.911415+0000","last_became_peered":"2023-11-08T08:18:42.911415+0000","last_unstale":"2023-11-08T08:18:44.272626+0000","last_undegraded":"2023-11-08T08:18:44.272626+0000","last_fullsized":"2023-11-08T08:18:44.272626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272344+0000","last_change":"2023-11-08T08:18:42.501133+0000","last_active":"2023-11-08T08:18:44.272344+0000","last_peered":"2023-11-08T08:18:44.272344+0000","last_clean":"2023-11-08T08:18:44.272344+0000","last_became_active":"2023-11-08T08:18:42.500842+0000","last_became_peered":"2023-11-08T08:18:42.500842+0000","last_unstale":"2023-11-08T08:18:44.272344+0000","last_undegraded":"2023-11-08T08:18:44.272344+0000","last_fullsized":"2023-11-08T08:18:44.272344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:05.271102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272942+0000","last_change":"2023-11-08T08:18:42.905081+0000","last_active":"2023-11-08T08:18:44.272942+0000","last_peered":"2023-11-08T08:18:44.272942+0000","last_clean":"2023-11-08T08:18:44.272942+0000","last_became_active":"2023-11-08T08:18:42.904781+0000","last_became_peered":"2023-11-08T08:18:42.904781+0000","last_unstale":"2023-11-08T08:18:44.272942+0000","last_undegraded":"2023-11-08T08:18:44.272942+0000","last_fullsized":"2023-11-08T08:18:44.272942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898380+0000","last_change":"2023-11-08T08:18:42.505921+0000","last_active":"2023-11-08T08:18:43.898380+0000","last_peered":"2023-11-08T08:18:43.898380+0000","last_clean":"2023-11-08T08:18:43.898380+0000","last_became_active":"2023-11-08T08:18:42.505591+0000","last_became_peered":"2023-11-08T08:18:42.505591+0000","last_unstale":"2023-11-08T08:18:43.898380+0000","last_undegraded":"2023-11-08T08:18:43.898380+0000","last_fullsized":"2023-11-08T08:18:43.898380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893978+0000","last_change":"2023-11-08T08:18:42.907871+0000","last_active":"2023-11-08T08:18:43.893978+0000","last_peered":"2023-11-08T08:18:43.893978+0000","last_clean":"2023-11-08T08:18:43.893978+0000","last_became_active":"2023-11-08T08:18:42.907170+0000","last_became_peered":"2023-11-08T08:18:42.907170+0000","last_unstale":"2023-11-08T08:18:43.893978+0000","last_undegraded":"2023-11-08T08:18:43.893978+0000","last_fullsized":"2023-11-08T08:18:43.893978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272471+0000","last_change":"2023-11-08T08:18:42.504796+0000","last_active":"2023-11-08T08:18:44.272471+0000","last_peered":"2023-11-08T08:18:44.272471+0000","last_clean":"2023-11-08T08:18:44.272471+0000","last_became_active":"2023-11-08T08:18:42.504498+0000","last_became_peered":"2023-11-08T08:18:42.504498+0000","last_unstale":"2023-11-08T08:18:44.272471+0000","last_undegraded":"2023-11-08T08:18:44.272471+0000","last_fullsized":"2023-11-08T08:18:44.272471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:02.270931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893958+0000","last_change":"2023-11-08T08:18:42.906660+0000","last_active":"2023-11-08T08:18:43.893958+0000","last_peered":"2023-11-08T08:18:43.893958+0000","last_clean":"2023-11-08T08:18:43.893958+0000","last_became_active":"2023-11-08T08:18:42.906410+0000","last_became_peered":"2023-11-08T08:18:42.906410+0000","last_unstale":"2023-11-08T08:18:43.893958+0000","last_undegraded":"2023-11-08T08:18:43.893958+0000","last_fullsized":"2023-11-08T08:18:43.893958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899130+0000","last_change":"2023-11-08T08:18:42.497356+0000","last_active":"2023-11-08T08:18:43.899130+0000","last_peered":"2023-11-08T08:18:43.899130+0000","last_clean":"2023-11-08T08:18:43.899130+0000","last_became_active":"2023-11-08T08:18:42.497056+0000","last_became_peered":"2023-11-08T08:18:42.497056+0000","last_unstale":"2023-11-08T08:18:43.899130+0000","last_undegraded":"2023-11-08T08:18:43.899130+0000","last_fullsized":"2023-11-08T08:18:43.899130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272096+0000","last_change":"2023-11-08T08:18:42.905346+0000","last_active":"2023-11-08T08:18:44.272096+0000","last_peered":"2023-11-08T08:18:44.272096+0000","last_clean":"2023-11-08T08:18:44.272096+0000","last_became_active":"2023-11-08T08:18:42.905050+0000","last_became_peered":"2023-11-08T08:18:42.905050+0000","last_unstale":"2023-11-08T08:18:44.272096+0000","last_undegraded":"2023-11-08T08:18:44.272096+0000","last_fullsized":"2023-11-08T08:18:44.272096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899328+0000","last_change":"2023-11-08T08:18:42.499349+0000","last_active":"2023-11-08T08:18:43.899328+0000","last_peered":"2023-11-08T08:18:43.899328+0000","last_clean":"2023-11-08T08:18:43.899328+0000","last_became_active":"2023-11-08T08:18:42.499050+0000","last_became_peered":"2023-11-08T08:18:42.499050+0000","last_unstale":"2023-11-08T08:18:43.899328+0000","last_undegraded":"2023-11-08T08:18:43.899328+0000","last_fullsized":"2023-11-08T08:18:43.899328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893668+0000","last_change":"2023-11-08T08:18:42.913391+0000","last_active":"2023-11-08T08:18:43.893668+0000","last_peered":"2023-11-08T08:18:43.893668+0000","last_clean":"2023-11-08T08:18:43.893668+0000","last_became_active":"2023-11-08T08:18:42.913124+0000","last_became_peered":"2023-11-08T08:18:42.913124+0000","last_unstale":"2023-11-08T08:18:43.893668+0000","last_undegraded":"2023-11-08T08:18:43.893668+0000","last_fullsized":"2023-11-08T08:18:43.893668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898165+0000","last_change":"2023-11-08T08:18:42.912208+0000","last_active":"2023-11-08T08:18:43.898165+0000","last_peered":"2023-11-08T08:18:43.898165+0000","last_clean":"2023-11-08T08:18:43.898165+0000","last_became_active":"2023-11-08T08:18:42.911928+0000","last_became_peered":"2023-11-08T08:18:42.911928+0000","last_unstale":"2023-11-08T08:18:43.898165+0000","last_undegraded":"2023-11-08T08:18:43.898165+0000","last_fullsized":"2023-11-08T08:18:43.898165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272255+0000","last_change":"2023-11-08T08:18:42.914191+0000","last_active":"2023-11-08T08:18:44.272255+0000","last_peered":"2023-11-08T08:18:44.272255+0000","last_clean":"2023-11-08T08:18:44.272255+0000","last_became_active":"2023-11-08T08:18:42.913919+0000","last_became_peered":"2023-11-08T08:18:42.913919+0000","last_unstale":"2023-11-08T08:18:44.272255+0000","last_undegraded":"2023-11-08T08:18:44.272255+0000","last_fullsized":"2023-11-08T08:18:44.272255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271708+0000","last_change":"2023-11-08T08:18:42.900802+0000","last_active":"2023-11-08T08:18:44.271708+0000","last_peered":"2023-11-08T08:18:44.271708+0000","last_clean":"2023-11-08T08:18:44.271708+0000","last_became_active":"2023-11-08T08:18:42.900538+0000","last_became_peered":"2023-11-08T08:18:42.900538+0000","last_unstale":"2023-11-08T08:18:44.271708+0000","last_undegraded":"2023-11-08T08:18:44.271708+0000","last_fullsized":"2023-11-08T08:18:44.271708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899133+0000","last_change":"2023-11-08T08:18:42.906606+0000","last_active":"2023-11-08T08:18:43.899133+0000","last_peered":"2023-11-08T08:18:43.899133+0000","last_clean":"2023-11-08T08:18:43.899133+0000","last_became_active":"2023-11-08T08:18:42.906279+0000","last_became_peered":"2023-11-08T08:18:42.906279+0000","last_unstale":"2023-11-08T08:18:43.899133+0000","last_undegraded":"2023-11-08T08:18:43.899133+0000","last_fullsized":"2023-11-08T08:18:43.899133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899035+0000","last_change":"2023-11-08T08:18:42.915689+0000","last_active":"2023-11-08T08:18:43.899035+0000","last_peered":"2023-11-08T08:18:43.899035+0000","last_clean":"2023-11-08T08:18:43.899035+0000","last_became_active":"2023-11-08T08:18:42.915121+0000","last_became_peered":"2023-11-08T08:18:42.915121+0000","last_unstale":"2023-11-08T08:18:43.899035+0000","last_undegraded":"2023-11-08T08:18:43.899035+0000","last_fullsized":"2023-11-08T08:18:43.899035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7888,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363952,"statfs":{"total":96636764160,"available":96628686848,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":73265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":74582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2759999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9064,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362776,"statfs":{"total":96636764160,"available":96627482624,"internally_reserved":0,"allocated":819200,"data_stored":666456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9048,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362792,"statfs":{"total":96636764160,"available":96627499008,"internally_reserved":0,"allocated":802816,"data_stored":665370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:18:46.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:18:47.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.110+0000 7fafbd425700 1 Processor -- start 2023-11-08T08:18:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.110+0000 7fafbd425700 1 -- start start 2023-11-08T08:18:47.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafbd425700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fafb806ab10 0x7fafb806aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafbd425700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fafb80cd5a0 0x7fafb8160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafbd425700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb80d1ee0 con 0x7fafb80b22a0 2023-11-08T08:18:47.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafbd425700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb806b420 con 0x7fafb806ab10 2023-11-08T08:18:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafbd425700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80cdb10 con 0x7fafb80cd5a0 2023-11-08T08:18:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fafb806ab10 0x7fafb806aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fafb806ab10 0x7fafb806aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39090/0 (socket says 172.21.15.116:39090) 2023-11-08T08:18:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 -- 172.21.15.116:0/223163355 learned_addr learned my addr 172.21.15.116:0/223163355 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb67fc700 1 --2- 172.21.15.116:0/223163355 >> v2:172.21.15.129:3300/0 conn(0x7fafb80cd5a0 0x7fafb8160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 -- 172.21.15.116:0/223163355 >> v2:172.21.15.129:3300/0 conn(0x7fafb80cd5a0 msgr2=0x7fafb8160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 --2- 172.21.15.116:0/223163355 >> v2:172.21.15.129:3300/0 conn(0x7fafb80cd5a0 0x7fafb8160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 -- 172.21.15.116:0/223163355 >> v1:172.21.15.129:6789/0 conn(0x7fafb80b22a0 legacy=0x7fafb8160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 -- 172.21.15.116:0/223163355 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80d1890 con 0x7fafb806ab10 2023-11-08T08:18:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb67fc700 1 --2- 172.21.15.116:0/223163355 >> v2:172.21.15.129:3300/0 conn(0x7fafb80cd5a0 0x7fafb8160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb6ffd700 1 --2- 172.21.15.116:0/223163355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb806aee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fafb80d0a60 tx=0x7fafac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e1e0f3bf6a5734 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb5ffb700 1 -- 172.21.15.116:0/223163355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac004230 con 0x7fafb806ab10 2023-11-08T08:18:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb5ffb700 1 -- 172.21.15.116:0/223163355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac004390 con 0x7fafb806ab10 2023-11-08T08:18:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.111+0000 7fafb5ffb700 1 -- 172.21.15.116:0/223163355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac0036a0 con 0x7fafb806ab10 2023-11-08T08:18:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 -- 172.21.15.116:0/223163355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 msgr2=0x7fafb806aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 --2- 172.21.15.116:0/223163355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb806aee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fafb80d0a60 tx=0x7fafac009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 -- 172.21.15.116:0/223163355 shutdown_connections 2023-11-08T08:18:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 --2- 172.21.15.116:0/223163355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb806aee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 --2- 172.21.15.116:0/223163355 >> v2:172.21.15.129:3300/0 conn(0x7fafb80cd5a0 0x7fafb8160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 -- 172.21.15.116:0/223163355 >> 172.21.15.116:0/223163355 conn(0x7fafb806c520 msgr2=0x7fafb806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 -- 172.21.15.116:0/223163355 shutdown_connections 2023-11-08T08:18:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.112+0000 7fafbd425700 1 -- 172.21.15.116:0/223163355 wait complete. 2023-11-08T08:18:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.113+0000 7fafbd425700 1 Processor -- start 2023-11-08T08:18:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.113+0000 7fafbd425700 1 -- start start 2023-11-08T08:18:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafbd425700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb80c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafbd425700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafb80b22a0 0x7fafb8153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafbd425700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fafb80cd5a0 0x7fafb8154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafbd425700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80c8430 con 0x7fafb806ab10 2023-11-08T08:18:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafbd425700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb81f2a10 con 0x7fafb80b22a0 2023-11-08T08:18:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafbd425700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80c7e70 con 0x7fafb80cd5a0 2023-11-08T08:18:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb80c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb80c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39096/0 (socket says 172.21.15.116:39096) 2023-11-08T08:18:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 -- 172.21.15.116:0/4249597351 learned_addr learned my addr 172.21.15.116:0/4249597351 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb67fc700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafb80b22a0 0x7fafb8153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 -- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fafb80cd5a0 msgr2=0x7fafb8154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb77fe700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fafb80cd5a0 0x7fafb8154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fafb80cd5a0 0x7fafb8154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 -- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafb80b22a0 msgr2=0x7fafb8153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafb80b22a0 0x7fafb8153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 -- 172.21.15.116:0/4249597351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac009550 con 0x7fafb806ab10 2023-11-08T08:18:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb67fc700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafb80b22a0 0x7fafb8153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb77fe700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fafb80cd5a0 0x7fafb8154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.114+0000 7fafb6ffd700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb80c7440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fafb806d0f0 tx=0x7fafac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.115+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac011630 con 0x7fafb806ab10 2023-11-08T08:18:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.115+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac011790 con 0x7fafb806ab10 2023-11-08T08:18:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.115+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb815cad0 con 0x7fafb806ab10 2023-11-08T08:18:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.115+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac011b40 con 0x7fafb806ab10 2023-11-08T08:18:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.115+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb80d1b70 con 0x7fafb806ab10 2023-11-08T08:18:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.116+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb80b1ec0 con 0x7fafb806ab10 2023-11-08T08:18:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.116+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fafac017020 con 0x7fafb806ab10 2023-11-08T08:18:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.116+0000 7faf9f7fe700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fafa40682f0 0x7fafa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.116+0000 7fafb67fc700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fafa40682f0 0x7fafa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.116+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fafac087290 con 0x7fafb806ab10 2023-11-08T08:18:47.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.117+0000 7fafb67fc700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fafa40682f0 0x7fafa406a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafb81f60e0 tx=0x7fafa8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.121+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fafac053af0 con 0x7fafb806ab10 2023-11-08T08:18:47.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.295+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafb80743f0 con 0x7fafa40682f0 2023-11-08T08:18:47.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.303+0000 7faf9f7fe700 1 -- 172.21.15.116:0/4249597351 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286927 (secure 0 0 0) 0x7fafb80743f0 con 0x7fafa40682f0 2023-11-08T08:18:47.306 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fafa40682f0 msgr2=0x7fafa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fafa40682f0 0x7fafa406a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafb81f60e0 tx=0x7fafa8008040 comp rx=0 tx=0).stop 2023-11-08T08:18:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 msgr2=0x7fafb80c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb80c7440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fafb806d0f0 tx=0x7fafac004750 comp rx=0 tx=0).stop 2023-11-08T08:18:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 shutdown_connections 2023-11-08T08:18:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fafb80cd5a0 0x7fafb8154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb806ab10 0x7fafb80c7440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fafa40682f0 0x7fafa406a7a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 --2- 172.21.15.116:0/4249597351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafb80b22a0 0x7fafb8153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 >> 172.21.15.116:0/4249597351 conn(0x7fafb806c520 msgr2=0x7fafb80b66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 shutdown_connections 2023-11-08T08:18:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.306+0000 7fafbd425700 1 -- 172.21.15.116:0/4249597351 wait complete. 2023-11-08T08:18:47.311 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:18:47.318 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-11-08T08:18:46.352511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65796,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":754908924,"statfs":{"total":773094113280,"available":773026738176,"internally_reserved":0,"allocated":2887680,"data_stored":1779623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.873937"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899181+0000","last_change":"2023-11-08T08:18:42.907639+0000","last_active":"2023-11-08T08:18:43.899181+0000","last_peered":"2023-11-08T08:18:43.899181+0000","last_clean":"2023-11-08T08:18:43.899181+0000","last_became_active":"2023-11-08T08:18:42.906449+0000","last_became_peered":"2023-11-08T08:18:42.906449+0000","last_unstale":"2023-11-08T08:18:43.899181+0000","last_undegraded":"2023-11-08T08:18:43.899181+0000","last_fullsized":"2023-11-08T08:18:43.899181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893783+0000","last_change":"2023-11-08T08:18:42.912729+0000","last_active":"2023-11-08T08:18:43.893783+0000","last_peered":"2023-11-08T08:18:43.893783+0000","last_clean":"2023-11-08T08:18:43.893783+0000","last_became_active":"2023-11-08T08:18:42.912482+0000","last_became_peered":"2023-11-08T08:18:42.912482+0000","last_unstale":"2023-11-08T08:18:43.893783+0000","last_undegraded":"2023-11-08T08:18:43.893783+0000","last_fullsized":"2023-11-08T08:18:43.893783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272081+0000","last_change":"2023-11-08T08:18:42.904767+0000","last_active":"2023-11-08T08:18:44.272081+0000","last_peered":"2023-11-08T08:18:44.272081+0000","last_clean":"2023-11-08T08:18:44.272081+0000","last_became_active":"2023-11-08T08:18:42.904512+0000","last_became_peered":"2023-11-08T08:18:42.904512+0000","last_unstale":"2023-11-08T08:18:44.272081+0000","last_undegraded":"2023-11-08T08:18:44.272081+0000","last_fullsized":"2023-11-08T08:18:44.272081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271750+0000","last_change":"2023-11-08T08:18:42.901716+0000","last_active":"2023-11-08T08:18:44.271750+0000","last_peered":"2023-11-08T08:18:44.271750+0000","last_clean":"2023-11-08T08:18:44.271750+0000","last_became_active":"2023-11-08T08:18:42.901477+0000","last_became_peered":"2023-11-08T08:18:42.901477+0000","last_unstale":"2023-11-08T08:18:44.271750+0000","last_undegraded":"2023-11-08T08:18:44.271750+0000","last_fullsized":"2023-11-08T08:18:44.271750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272162+0000","last_change":"2023-11-08T08:18:42.906339+0000","last_active":"2023-11-08T08:18:44.272162+0000","last_peered":"2023-11-08T08:18:44.272162+0000","last_clean":"2023-11-08T08:18:44.272162+0000","last_became_active":"2023-11-08T08:18:42.906082+0000","last_became_peered":"2023-11-08T08:18:42.906082+0000","last_unstale":"2023-11-08T08:18:44.272162+0000","last_undegraded":"2023-11-08T08:18:44.272162+0000","last_fullsized":"2023-11-08T08:18:44.272162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272523+0000","last_change":"2023-11-08T08:18:42.907735+0000","last_active":"2023-11-08T08:18:44.272523+0000","last_peered":"2023-11-08T08:18:44.272523+0000","last_clean":"2023-11-08T08:18:44.272523+0000","last_became_active":"2023-11-08T08:18:42.907414+0000","last_became_peered":"2023-11-08T08:18:42.907414+0000","last_unstale":"2023-11-08T08:18:44.272523+0000","last_undegraded":"2023-11-08T08:18:44.272523+0000","last_fullsized":"2023-11-08T08:18:44.272523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271898+0000","last_change":"2023-11-08T08:18:42.904968+0000","last_active":"2023-11-08T08:18:44.271898+0000","last_peered":"2023-11-08T08:18:44.271898+0000","last_clean":"2023-11-08T08:18:44.271898+0000","last_became_active":"2023-11-08T08:18:42.904586+0000","last_became_peered":"2023-11-08T08:18:42.904586+0000","last_unstale":"2023-11-08T08:18:44.271898+0000","last_undegraded":"2023-11-08T08:18:44.271898+0000","last_fullsized":"2023-11-08T08:18:44.271898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:08.733824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898246+0000","last_change":"2023-11-08T08:18:42.898509+0000","last_active":"2023-11-08T08:18:43.898246+0000","last_peered":"2023-11-08T08:18:43.898246+0000","last_clean":"2023-11-08T08:18:43.898246+0000","last_became_active":"2023-11-08T08:18:42.898230+0000","last_became_peered":"2023-11-08T08:18:42.898230+0000","last_unstale":"2023-11-08T08:18:43.898246+0000","last_undegraded":"2023-11-08T08:18:43.898246+0000","last_fullsized":"2023-11-08T08:18:43.898246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273075+0000","last_change":"2023-11-08T08:18:42.906809+0000","last_active":"2023-11-08T08:18:44.273075+0000","last_peered":"2023-11-08T08:18:44.273075+0000","last_clean":"2023-11-08T08:18:44.273075+0000","last_became_active":"2023-11-08T08:18:42.906517+0000","last_became_peered":"2023-11-08T08:18:42.906517+0000","last_unstale":"2023-11-08T08:18:44.273075+0000","last_undegraded":"2023-11-08T08:18:44.273075+0000","last_fullsized":"2023-11-08T08:18:44.273075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898585+0000","last_change":"2023-11-08T08:18:42.906651+0000","last_active":"2023-11-08T08:18:43.898585+0000","last_peered":"2023-11-08T08:18:43.898585+0000","last_clean":"2023-11-08T08:18:43.898585+0000","last_became_active":"2023-11-08T08:18:42.906379+0000","last_became_peered":"2023-11-08T08:18:42.906379+0000","last_unstale":"2023-11-08T08:18:43.898585+0000","last_undegraded":"2023-11-08T08:18:43.898585+0000","last_fullsized":"2023-11-08T08:18:43.898585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898449+0000","last_change":"2023-11-08T08:18:42.898832+0000","last_active":"2023-11-08T08:18:43.898449+0000","last_peered":"2023-11-08T08:18:43.898449+0000","last_clean":"2023-11-08T08:18:43.898449+0000","last_became_active":"2023-11-08T08:18:42.898581+0000","last_became_peered":"2023-11-08T08:18:42.898581+0000","last_unstale":"2023-11-08T08:18:43.898449+0000","last_undegraded":"2023-11-08T08:18:43.898449+0000","last_fullsized":"2023-11-08T08:18:43.898449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898948+0000","last_change":"2023-11-08T08:18:42.911043+0000","last_active":"2023-11-08T08:18:43.898948+0000","last_peered":"2023-11-08T08:18:43.898948+0000","last_clean":"2023-11-08T08:18:43.898948+0000","last_became_active":"2023-11-08T08:18:42.910756+0000","last_became_peered":"2023-11-08T08:18:42.910756+0000","last_unstale":"2023-11-08T08:18:43.898948+0000","last_undegraded":"2023-11-08T08:18:43.898948+0000","last_fullsized":"2023-11-08T08:18:43.898948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272630+0000","last_change":"2023-11-08T08:18:42.915118+0000","last_active":"2023-11-08T08:18:44.272630+0000","last_peered":"2023-11-08T08:18:44.272630+0000","last_clean":"2023-11-08T08:18:44.272630+0000","last_became_active":"2023-11-08T08:18:42.914854+0000","last_became_peered":"2023-11-08T08:18:42.914854+0000","last_unstale":"2023-11-08T08:18:44.272630+0000","last_undegraded":"2023-11-08T08:18:44.272630+0000","last_fullsized":"2023-11-08T08:18:44.272630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898604+0000","last_change":"2023-11-08T08:18:42.900369+0000","last_active":"2023-11-08T08:18:43.898604+0000","last_peered":"2023-11-08T08:18:43.898604+0000","last_clean":"2023-11-08T08:18:43.898604+0000","last_became_active":"2023-11-08T08:18:42.900121+0000","last_became_peered":"2023-11-08T08:18:42.900121+0000","last_unstale":"2023-11-08T08:18:43.898604+0000","last_undegraded":"2023-11-08T08:18:43.898604+0000","last_fullsized":"2023-11-08T08:18:43.898604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271716+0000","last_change":"2023-11-08T08:18:42.912567+0000","last_active":"2023-11-08T08:18:44.271716+0000","last_peered":"2023-11-08T08:18:44.271716+0000","last_clean":"2023-11-08T08:18:44.271716+0000","last_became_active":"2023-11-08T08:18:42.912299+0000","last_became_peered":"2023-11-08T08:18:42.912299+0000","last_unstale":"2023-11-08T08:18:44.271716+0000","last_undegraded":"2023-11-08T08:18:44.271716+0000","last_fullsized":"2023-11-08T08:18:44.271716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271758+0000","last_change":"2023-11-08T08:18:42.902375+0000","last_active":"2023-11-08T08:18:44.271758+0000","last_peered":"2023-11-08T08:18:44.271758+0000","last_clean":"2023-11-08T08:18:44.271758+0000","last_became_active":"2023-11-08T08:18:42.901640+0000","last_became_peered":"2023-11-08T08:18:42.901640+0000","last_unstale":"2023-11-08T08:18:44.271758+0000","last_undegraded":"2023-11-08T08:18:44.271758+0000","last_fullsized":"2023-11-08T08:18:44.271758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272211+0000","last_change":"2023-11-08T08:18:42.905755+0000","last_active":"2023-11-08T08:18:44.272211+0000","last_peered":"2023-11-08T08:18:44.272211+0000","last_clean":"2023-11-08T08:18:44.272211+0000","last_became_active":"2023-11-08T08:18:42.905498+0000","last_became_peered":"2023-11-08T08:18:42.905498+0000","last_unstale":"2023-11-08T08:18:44.272211+0000","last_undegraded":"2023-11-08T08:18:44.272211+0000","last_fullsized":"2023-11-08T08:18:44.272211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272018+0000","last_change":"2023-11-08T08:18:42.900992+0000","last_active":"2023-11-08T08:18:44.272018+0000","last_peered":"2023-11-08T08:18:44.272018+0000","last_clean":"2023-11-08T08:18:44.272018+0000","last_became_active":"2023-11-08T08:18:42.900747+0000","last_became_peered":"2023-11-08T08:18:42.900747+0000","last_unstale":"2023-11-08T08:18:44.272018+0000","last_undegraded":"2023-11-08T08:18:44.272018+0000","last_fullsized":"2023-11-08T08:18:44.272018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899009+0000","last_change":"2023-11-08T08:18:42.910975+0000","last_active":"2023-11-08T08:18:43.899009+0000","last_peered":"2023-11-08T08:18:43.899009+0000","last_clean":"2023-11-08T08:18:43.899009+0000","last_became_active":"2023-11-08T08:18:42.910701+0000","last_became_peered":"2023-11-08T08:18:42.910701+0000","last_unstale":"2023-11-08T08:18:43.899009+0000","last_undegraded":"2023-11-08T08:18:43.899009+0000","last_fullsized":"2023-11-08T08:18:43.899009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898283+0000","last_change":"2023-11-08T08:18:42.912881+0000","last_active":"2023-11-08T08:18:43.898283+0000","last_peered":"2023-11-08T08:18:43.898283+0000","last_clean":"2023-11-08T08:18:43.898283+0000","last_became_active":"2023-11-08T08:18:42.912660+0000","last_became_peered":"2023-11-08T08:18:42.912660+0000","last_unstale":"2023-11-08T08:18:43.898283+0000","last_undegraded":"2023-11-08T08:18:43.898283+0000","last_fullsized":"2023-11-08T08:18:43.898283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898341+0000","last_change":"2023-11-08T08:18:42.901149+0000","last_active":"2023-11-08T08:18:43.898341+0000","last_peered":"2023-11-08T08:18:43.898341+0000","last_clean":"2023-11-08T08:18:43.898341+0000","last_became_active":"2023-11-08T08:18:42.900728+0000","last_became_peered":"2023-11-08T08:18:42.900728+0000","last_unstale":"2023-11-08T08:18:43.898341+0000","last_undegraded":"2023-11-08T08:18:43.898341+0000","last_fullsized":"2023-11-08T08:18:43.898341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272020+0000","last_change":"2023-11-08T08:18:42.911764+0000","last_active":"2023-11-08T08:18:44.272020+0000","last_peered":"2023-11-08T08:18:44.272020+0000","last_clean":"2023-11-08T08:18:44.272020+0000","last_became_active":"2023-11-08T08:18:42.911477+0000","last_became_peered":"2023-11-08T08:18:42.911477+0000","last_unstale":"2023-11-08T08:18:44.272020+0000","last_undegraded":"2023-11-08T08:18:44.272020+0000","last_fullsized":"2023-11-08T08:18:44.272020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272570+0000","last_change":"2023-11-08T08:18:42.909892+0000","last_active":"2023-11-08T08:18:44.272570+0000","last_peered":"2023-11-08T08:18:44.272570+0000","last_clean":"2023-11-08T08:18:44.272570+0000","last_became_active":"2023-11-08T08:18:42.909646+0000","last_became_peered":"2023-11-08T08:18:42.909646+0000","last_unstale":"2023-11-08T08:18:44.272570+0000","last_undegraded":"2023-11-08T08:18:44.272570+0000","last_fullsized":"2023-11-08T08:18:44.272570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898218+0000","last_change":"2023-11-08T08:18:42.897278+0000","last_active":"2023-11-08T08:18:43.898218+0000","last_peered":"2023-11-08T08:18:43.898218+0000","last_clean":"2023-11-08T08:18:43.898218+0000","last_became_active":"2023-11-08T08:18:42.896923+0000","last_became_peered":"2023-11-08T08:18:42.896923+0000","last_unstale":"2023-11-08T08:18:43.898218+0000","last_undegraded":"2023-11-08T08:18:43.898218+0000","last_fullsized":"2023-11-08T08:18:43.898218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273144+0000","last_change":"2023-11-08T08:18:42.909514+0000","last_active":"2023-11-08T08:18:44.273144+0000","last_peered":"2023-11-08T08:18:44.273144+0000","last_clean":"2023-11-08T08:18:44.273144+0000","last_became_active":"2023-11-08T08:18:42.909246+0000","last_became_peered":"2023-11-08T08:18:42.909246+0000","last_unstale":"2023-11-08T08:18:44.273144+0000","last_undegraded":"2023-11-08T08:18:44.273144+0000","last_fullsized":"2023-11-08T08:18:44.273144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272924+0000","last_change":"2023-11-08T08:18:42.912860+0000","last_active":"2023-11-08T08:18:44.272924+0000","last_peered":"2023-11-08T08:18:44.272924+0000","last_clean":"2023-11-08T08:18:44.272924+0000","last_became_active":"2023-11-08T08:18:42.912613+0000","last_became_peered":"2023-11-08T08:18:42.912613+0000","last_unstale":"2023-11-08T08:18:44.272924+0000","last_undegraded":"2023-11-08T08:18:44.272924+0000","last_fullsized":"2023-11-08T08:18:44.272924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272694+0000","last_change":"2023-11-08T08:18:42.898517+0000","last_active":"2023-11-08T08:18:44.272694+0000","last_peered":"2023-11-08T08:18:44.272694+0000","last_clean":"2023-11-08T08:18:44.272694+0000","last_became_active":"2023-11-08T08:18:42.898253+0000","last_became_peered":"2023-11-08T08:18:42.898253+0000","last_unstale":"2023-11-08T08:18:44.272694+0000","last_undegraded":"2023-11-08T08:18:44.272694+0000","last_fullsized":"2023-11-08T08:18:44.272694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:56:57.876135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898142+0000","last_change":"2023-11-08T08:18:42.501016+0000","last_active":"2023-11-08T08:18:43.898142+0000","last_peered":"2023-11-08T08:18:43.898142+0000","last_clean":"2023-11-08T08:18:43.898142+0000","last_became_active":"2023-11-08T08:18:42.499597+0000","last_became_peered":"2023-11-08T08:18:42.499597+0000","last_unstale":"2023-11-08T08:18:43.898142+0000","last_undegraded":"2023-11-08T08:18:43.898142+0000","last_fullsized":"2023-11-08T08:18:43.898142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272484+0000","last_change":"2023-11-08T08:18:42.907418+0000","last_active":"2023-11-08T08:18:44.272484+0000","last_peered":"2023-11-08T08:18:44.272484+0000","last_clean":"2023-11-08T08:18:44.272484+0000","last_became_active":"2023-11-08T08:18:42.907045+0000","last_became_peered":"2023-11-08T08:18:42.907045+0000","last_unstale":"2023-11-08T08:18:44.272484+0000","last_undegraded":"2023-11-08T08:18:44.272484+0000","last_fullsized":"2023-11-08T08:18:44.272484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272507+0000","last_change":"2023-11-08T08:18:42.497426+0000","last_active":"2023-11-08T08:18:44.272507+0000","last_peered":"2023-11-08T08:18:44.272507+0000","last_clean":"2023-11-08T08:18:44.272507+0000","last_became_active":"2023-11-08T08:18:42.497130+0000","last_became_peered":"2023-11-08T08:18:42.497130+0000","last_unstale":"2023-11-08T08:18:44.272507+0000","last_undegraded":"2023-11-08T08:18:44.272507+0000","last_fullsized":"2023-11-08T08:18:44.272507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272906+0000","last_change":"2023-11-08T08:18:42.905727+0000","last_active":"2023-11-08T08:18:44.272906+0000","last_peered":"2023-11-08T08:18:44.272906+0000","last_clean":"2023-11-08T08:18:44.272906+0000","last_became_active":"2023-11-08T08:18:42.905467+0000","last_became_peered":"2023-11-08T08:18:42.905467+0000","last_unstale":"2023-11-08T08:18:44.272906+0000","last_undegraded":"2023-11-08T08:18:44.272906+0000","last_fullsized":"2023-11-08T08:18:44.272906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898639+0000","last_change":"2023-11-08T08:18:42.502963+0000","last_active":"2023-11-08T08:18:43.898639+0000","last_peered":"2023-11-08T08:18:43.898639+0000","last_clean":"2023-11-08T08:18:43.898639+0000","last_became_active":"2023-11-08T08:18:42.502702+0000","last_became_peered":"2023-11-08T08:18:42.502702+0000","last_unstale":"2023-11-08T08:18:43.898639+0000","last_undegraded":"2023-11-08T08:18:43.898639+0000","last_fullsized":"2023-11-08T08:18:43.898639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272590+0000","last_change":"2023-11-08T08:18:42.904909+0000","last_active":"2023-11-08T08:18:44.272590+0000","last_peered":"2023-11-08T08:18:44.272590+0000","last_clean":"2023-11-08T08:18:44.272590+0000","last_became_active":"2023-11-08T08:18:42.903891+0000","last_became_peered":"2023-11-08T08:18:42.903891+0000","last_unstale":"2023-11-08T08:18:44.272590+0000","last_undegraded":"2023-11-08T08:18:44.272590+0000","last_fullsized":"2023-11-08T08:18:44.272590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:21.735174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899300+0000","last_change":"2023-11-08T08:18:42.498275+0000","last_active":"2023-11-08T08:18:43.899300+0000","last_peered":"2023-11-08T08:18:43.899300+0000","last_clean":"2023-11-08T08:18:43.899300+0000","last_became_active":"2023-11-08T08:18:42.497975+0000","last_became_peered":"2023-11-08T08:18:42.497975+0000","last_unstale":"2023-11-08T08:18:43.899300+0000","last_undegraded":"2023-11-08T08:18:43.899300+0000","last_fullsized":"2023-11-08T08:18:43.899300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271740+0000","last_change":"2023-11-08T08:18:42.900693+0000","last_active":"2023-11-08T08:18:44.271740+0000","last_peered":"2023-11-08T08:18:44.271740+0000","last_clean":"2023-11-08T08:18:44.271740+0000","last_became_active":"2023-11-08T08:18:42.900407+0000","last_became_peered":"2023-11-08T08:18:42.900407+0000","last_unstale":"2023-11-08T08:18:44.271740+0000","last_undegraded":"2023-11-08T08:18:44.271740+0000","last_fullsized":"2023-11-08T08:18:44.271740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894013+0000","last_change":"2023-11-08T08:18:42.502815+0000","last_active":"2023-11-08T08:18:43.894013+0000","last_peered":"2023-11-08T08:18:43.894013+0000","last_clean":"2023-11-08T08:18:43.894013+0000","last_became_active":"2023-11-08T08:18:42.502329+0000","last_became_peered":"2023-11-08T08:18:42.502329+0000","last_unstale":"2023-11-08T08:18:43.894013+0000","last_undegraded":"2023-11-08T08:18:43.894013+0000","last_fullsized":"2023-11-08T08:18:43.894013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272456+0000","last_change":"2023-11-08T08:18:42.909334+0000","last_active":"2023-11-08T08:18:44.272456+0000","last_peered":"2023-11-08T08:18:44.272456+0000","last_clean":"2023-11-08T08:18:44.272456+0000","last_became_active":"2023-11-08T08:18:42.907416+0000","last_became_peered":"2023-11-08T08:18:42.907416+0000","last_unstale":"2023-11-08T08:18:44.272456+0000","last_undegraded":"2023-11-08T08:18:44.272456+0000","last_fullsized":"2023-11-08T08:18:44.272456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272167+0000","last_change":"2023-11-08T08:18:42.504744+0000","last_active":"2023-11-08T08:18:44.272167+0000","last_peered":"2023-11-08T08:18:44.272167+0000","last_clean":"2023-11-08T08:18:44.272167+0000","last_became_active":"2023-11-08T08:18:42.504438+0000","last_became_peered":"2023-11-08T08:18:42.504438+0000","last_unstale":"2023-11-08T08:18:44.272167+0000","last_undegraded":"2023-11-08T08:18:44.272167+0000","last_fullsized":"2023-11-08T08:18:44.272167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272646+0000","last_change":"2023-11-08T08:18:42.910922+0000","last_active":"2023-11-08T08:18:44.272646+0000","last_peered":"2023-11-08T08:18:44.272646+0000","last_clean":"2023-11-08T08:18:44.272646+0000","last_became_active":"2023-11-08T08:18:42.910578+0000","last_became_peered":"2023-11-08T08:18:42.910578+0000","last_unstale":"2023-11-08T08:18:44.272646+0000","last_undegraded":"2023-11-08T08:18:44.272646+0000","last_fullsized":"2023-11-08T08:18:44.272646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272486+0000","last_change":"2023-11-08T08:18:42.497349+0000","last_active":"2023-11-08T08:18:44.272486+0000","last_peered":"2023-11-08T08:18:44.272486+0000","last_clean":"2023-11-08T08:18:44.272486+0000","last_became_active":"2023-11-08T08:18:42.497038+0000","last_became_peered":"2023-11-08T08:18:42.497038+0000","last_unstale":"2023-11-08T08:18:44.272486+0000","last_undegraded":"2023-11-08T08:18:44.272486+0000","last_fullsized":"2023-11-08T08:18:44.272486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:12.090061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894075+0000","last_change":"2023-11-08T08:18:42.905780+0000","last_active":"2023-11-08T08:18:43.894075+0000","last_peered":"2023-11-08T08:18:43.894075+0000","last_clean":"2023-11-08T08:18:43.894075+0000","last_became_active":"2023-11-08T08:18:42.905487+0000","last_became_peered":"2023-11-08T08:18:42.905487+0000","last_unstale":"2023-11-08T08:18:43.894075+0000","last_undegraded":"2023-11-08T08:18:43.894075+0000","last_fullsized":"2023-11-08T08:18:43.894075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272417+0000","last_change":"2023-11-08T08:18:42.504787+0000","last_active":"2023-11-08T08:18:44.272417+0000","last_peered":"2023-11-08T08:18:44.272417+0000","last_clean":"2023-11-08T08:18:44.272417+0000","last_became_active":"2023-11-08T08:18:42.504477+0000","last_became_peered":"2023-11-08T08:18:42.504477+0000","last_unstale":"2023-11-08T08:18:44.272417+0000","last_undegraded":"2023-11-08T08:18:44.272417+0000","last_fullsized":"2023-11-08T08:18:44.272417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:38.511975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272076+0000","last_change":"2023-11-08T08:18:42.913147+0000","last_active":"2023-11-08T08:18:44.272076+0000","last_peered":"2023-11-08T08:18:44.272076+0000","last_clean":"2023-11-08T08:18:44.272076+0000","last_became_active":"2023-11-08T08:18:42.912855+0000","last_became_peered":"2023-11-08T08:18:42.912855+0000","last_unstale":"2023-11-08T08:18:44.272076+0000","last_undegraded":"2023-11-08T08:18:44.272076+0000","last_fullsized":"2023-11-08T08:18:44.272076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272506+0000","last_change":"2023-11-08T08:18:42.504414+0000","last_active":"2023-11-08T08:18:44.272506+0000","last_peered":"2023-11-08T08:18:44.272506+0000","last_clean":"2023-11-08T08:18:44.272506+0000","last_became_active":"2023-11-08T08:18:42.504141+0000","last_became_peered":"2023-11-08T08:18:42.504141+0000","last_unstale":"2023-11-08T08:18:44.272506+0000","last_undegraded":"2023-11-08T08:18:44.272506+0000","last_fullsized":"2023-11-08T08:18:44.272506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272647+0000","last_change":"2023-11-08T08:18:42.908975+0000","last_active":"2023-11-08T08:18:44.272647+0000","last_peered":"2023-11-08T08:18:44.272647+0000","last_clean":"2023-11-08T08:18:44.272647+0000","last_became_active":"2023-11-08T08:18:42.908137+0000","last_became_peered":"2023-11-08T08:18:42.908137+0000","last_unstale":"2023-11-08T08:18:44.272647+0000","last_undegraded":"2023-11-08T08:18:44.272647+0000","last_fullsized":"2023-11-08T08:18:44.272647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:35:59.915321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.495958+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.495711+0000","last_became_peered":"2023-11-08T08:18:42.495711+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:04.545064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899087+0000","last_change":"2023-11-08T08:18:42.907135+0000","last_active":"2023-11-08T08:18:43.899087+0000","last_peered":"2023-11-08T08:18:43.899087+0000","last_clean":"2023-11-08T08:18:43.899087+0000","last_became_active":"2023-11-08T08:18:42.906893+0000","last_became_peered":"2023-11-08T08:18:42.906893+0000","last_unstale":"2023-11-08T08:18:43.899087+0000","last_undegraded":"2023-11-08T08:18:43.899087+0000","last_fullsized":"2023-11-08T08:18:43.899087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898115+0000","last_change":"2023-11-08T08:18:42.505141+0000","last_active":"2023-11-08T08:18:43.898115+0000","last_peered":"2023-11-08T08:18:43.898115+0000","last_clean":"2023-11-08T08:18:43.898115+0000","last_became_active":"2023-11-08T08:18:42.500393+0000","last_became_peered":"2023-11-08T08:18:42.500393+0000","last_unstale":"2023-11-08T08:18:43.898115+0000","last_undegraded":"2023-11-08T08:18:43.898115+0000","last_fullsized":"2023-11-08T08:18:43.898115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899261+0000","last_change":"2023-11-08T08:18:42.908933+0000","last_active":"2023-11-08T08:18:43.899261+0000","last_peered":"2023-11-08T08:18:43.899261+0000","last_clean":"2023-11-08T08:18:43.899261+0000","last_became_active":"2023-11-08T08:18:42.908643+0000","last_became_peered":"2023-11-08T08:18:42.908643+0000","last_unstale":"2023-11-08T08:18:43.899261+0000","last_undegraded":"2023-11-08T08:18:43.899261+0000","last_fullsized":"2023-11-08T08:18:43.899261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272143+0000","last_change":"2023-11-08T08:18:42.506315+0000","last_active":"2023-11-08T08:18:44.272143+0000","last_peered":"2023-11-08T08:18:44.272143+0000","last_clean":"2023-11-08T08:18:44.272143+0000","last_became_active":"2023-11-08T08:18:42.506044+0000","last_became_peered":"2023-11-08T08:18:42.506044+0000","last_unstale":"2023-11-08T08:18:44.272143+0000","last_undegraded":"2023-11-08T08:18:44.272143+0000","last_fullsized":"2023-11-08T08:18:44.272143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271908+0000","last_change":"2023-11-08T08:18:42.908490+0000","last_active":"2023-11-08T08:18:44.271908+0000","last_peered":"2023-11-08T08:18:44.271908+0000","last_clean":"2023-11-08T08:18:44.271908+0000","last_became_active":"2023-11-08T08:18:42.908302+0000","last_became_peered":"2023-11-08T08:18:42.908302+0000","last_unstale":"2023-11-08T08:18:44.271908+0000","last_undegraded":"2023-11-08T08:18:44.271908+0000","last_fullsized":"2023-11-08T08:18:44.271908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893984+0000","last_change":"2023-11-08T08:18:42.504388+0000","last_active":"2023-11-08T08:18:43.893984+0000","last_peered":"2023-11-08T08:18:43.893984+0000","last_clean":"2023-11-08T08:18:43.893984+0000","last_became_active":"2023-11-08T08:18:42.504085+0000","last_became_peered":"2023-11-08T08:18:42.504085+0000","last_unstale":"2023-11-08T08:18:43.893984+0000","last_undegraded":"2023-11-08T08:18:43.893984+0000","last_fullsized":"2023-11-08T08:18:43.893984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898126+0000","last_change":"2023-11-08T08:18:42.901943+0000","last_active":"2023-11-08T08:18:43.898126+0000","last_peered":"2023-11-08T08:18:43.898126+0000","last_clean":"2023-11-08T08:18:43.898126+0000","last_became_active":"2023-11-08T08:18:42.901687+0000","last_became_peered":"2023-11-08T08:18:42.901687+0000","last_unstale":"2023-11-08T08:18:43.898126+0000","last_undegraded":"2023-11-08T08:18:43.898126+0000","last_fullsized":"2023-11-08T08:18:43.898126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272582+0000","last_change":"2023-11-08T08:18:42.502289+0000","last_active":"2023-11-08T08:18:44.272582+0000","last_peered":"2023-11-08T08:18:44.272582+0000","last_clean":"2023-11-08T08:18:44.272582+0000","last_became_active":"2023-11-08T08:18:42.499404+0000","last_became_peered":"2023-11-08T08:18:42.499404+0000","last_unstale":"2023-11-08T08:18:44.272582+0000","last_undegraded":"2023-11-08T08:18:44.272582+0000","last_fullsized":"2023-11-08T08:18:44.272582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:43.337457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893816+0000","last_change":"2023-11-08T08:18:42.905468+0000","last_active":"2023-11-08T08:18:43.893816+0000","last_peered":"2023-11-08T08:18:43.893816+0000","last_clean":"2023-11-08T08:18:43.893816+0000","last_became_active":"2023-11-08T08:18:42.905203+0000","last_became_peered":"2023-11-08T08:18:42.905203+0000","last_unstale":"2023-11-08T08:18:43.893816+0000","last_undegraded":"2023-11-08T08:18:43.893816+0000","last_fullsized":"2023-11-08T08:18:43.893816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.503204+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.501193+0000","last_became_peered":"2023-11-08T08:18:42.501193+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899051+0000","last_change":"2023-11-08T08:18:42.911979+0000","last_active":"2023-11-08T08:18:43.899051+0000","last_peered":"2023-11-08T08:18:43.899051+0000","last_clean":"2023-11-08T08:18:43.899051+0000","last_became_active":"2023-11-08T08:18:42.911070+0000","last_became_peered":"2023-11-08T08:18:42.911070+0000","last_unstale":"2023-11-08T08:18:43.899051+0000","last_undegraded":"2023-11-08T08:18:43.899051+0000","last_fullsized":"2023-11-08T08:18:43.899051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272150+0000","last_change":"2023-11-08T08:18:42.503625+0000","last_active":"2023-11-08T08:18:44.272150+0000","last_peered":"2023-11-08T08:18:44.272150+0000","last_clean":"2023-11-08T08:18:44.272150+0000","last_became_active":"2023-11-08T08:18:42.503351+0000","last_became_peered":"2023-11-08T08:18:42.503351+0000","last_unstale":"2023-11-08T08:18:44.272150+0000","last_undegraded":"2023-11-08T08:18:44.272150+0000","last_fullsized":"2023-11-08T08:18:44.272150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271896+0000","last_change":"2023-11-08T08:18:42.910859+0000","last_active":"2023-11-08T08:18:44.271896+0000","last_peered":"2023-11-08T08:18:44.271896+0000","last_clean":"2023-11-08T08:18:44.271896+0000","last_became_active":"2023-11-08T08:18:42.910598+0000","last_became_peered":"2023-11-08T08:18:42.910598+0000","last_unstale":"2023-11-08T08:18:44.271896+0000","last_undegraded":"2023-11-08T08:18:44.271896+0000","last_fullsized":"2023-11-08T08:18:44.271896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272298+0000","last_change":"2023-11-08T08:18:42.499085+0000","last_active":"2023-11-08T08:18:44.272298+0000","last_peered":"2023-11-08T08:18:44.272298+0000","last_clean":"2023-11-08T08:18:44.272298+0000","last_became_active":"2023-11-08T08:18:42.498750+0000","last_became_peered":"2023-11-08T08:18:42.498750+0000","last_unstale":"2023-11-08T08:18:44.272298+0000","last_undegraded":"2023-11-08T08:18:44.272298+0000","last_fullsized":"2023-11-08T08:18:44.272298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272549+0000","last_change":"2023-11-08T08:18:42.903923+0000","last_active":"2023-11-08T08:18:44.272549+0000","last_peered":"2023-11-08T08:18:44.272549+0000","last_clean":"2023-11-08T08:18:44.272549+0000","last_became_active":"2023-11-08T08:18:42.903518+0000","last_became_peered":"2023-11-08T08:18:42.903518+0000","last_unstale":"2023-11-08T08:18:44.272549+0000","last_undegraded":"2023-11-08T08:18:44.272549+0000","last_fullsized":"2023-11-08T08:18:44.272549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:51:16.185661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272139+0000","last_change":"2023-11-08T08:18:42.496450+0000","last_active":"2023-11-08T08:18:44.272139+0000","last_peered":"2023-11-08T08:18:44.272139+0000","last_clean":"2023-11-08T08:18:44.272139+0000","last_became_active":"2023-11-08T08:18:42.496197+0000","last_became_peered":"2023-11-08T08:18:42.496197+0000","last_unstale":"2023-11-08T08:18:44.272139+0000","last_undegraded":"2023-11-08T08:18:44.272139+0000","last_fullsized":"2023-11-08T08:18:44.272139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893897+0000","last_change":"2023-11-08T08:18:42.906952+0000","last_active":"2023-11-08T08:18:43.893897+0000","last_peered":"2023-11-08T08:18:43.893897+0000","last_clean":"2023-11-08T08:18:43.893897+0000","last_became_active":"2023-11-08T08:18:42.906699+0000","last_became_peered":"2023-11-08T08:18:42.906699+0000","last_unstale":"2023-11-08T08:18:43.893897+0000","last_undegraded":"2023-11-08T08:18:43.893897+0000","last_fullsized":"2023-11-08T08:18:43.893897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271876+0000","last_change":"2023-11-08T08:18:42.496216+0000","last_active":"2023-11-08T08:18:44.271876+0000","last_peered":"2023-11-08T08:18:44.271876+0000","last_clean":"2023-11-08T08:18:44.271876+0000","last_became_active":"2023-11-08T08:18:42.495940+0000","last_became_peered":"2023-11-08T08:18:42.495940+0000","last_unstale":"2023-11-08T08:18:44.271876+0000","last_undegraded":"2023-11-08T08:18:44.271876+0000","last_fullsized":"2023-11-08T08:18:44.271876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272347+0000","last_change":"2023-11-08T08:18:42.916881+0000","last_active":"2023-11-08T08:18:44.272347+0000","last_peered":"2023-11-08T08:18:44.272347+0000","last_clean":"2023-11-08T08:18:44.272347+0000","last_became_active":"2023-11-08T08:18:42.916593+0000","last_became_peered":"2023-11-08T08:18:42.916593+0000","last_unstale":"2023-11-08T08:18:44.272347+0000","last_undegraded":"2023-11-08T08:18:44.272347+0000","last_fullsized":"2023-11-08T08:18:44.272347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:09.598915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273120+0000","last_change":"2023-11-08T08:18:42.502915+0000","last_active":"2023-11-08T08:18:44.273120+0000","last_peered":"2023-11-08T08:18:44.273120+0000","last_clean":"2023-11-08T08:18:44.273120+0000","last_became_active":"2023-11-08T08:18:42.502675+0000","last_became_peered":"2023-11-08T08:18:42.502675+0000","last_unstale":"2023-11-08T08:18:44.273120+0000","last_undegraded":"2023-11-08T08:18:44.273120+0000","last_fullsized":"2023-11-08T08:18:44.273120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:08.593928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.891523+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:18:42.891523+0000","last_peered":"2023-11-08T08:18:42.891523+0000","last_clean":"2023-11-08T08:18:42.891523+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:18:42.891523+0000","last_undegraded":"2023-11-08T08:18:42.891523+0000","last_fullsized":"2023-11-08T08:18:42.891523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271984+0000","last_change":"2023-11-08T08:18:42.908748+0000","last_active":"2023-11-08T08:18:44.271984+0000","last_peered":"2023-11-08T08:18:44.271984+0000","last_clean":"2023-11-08T08:18:44.271984+0000","last_became_active":"2023-11-08T08:18:42.908419+0000","last_became_peered":"2023-11-08T08:18:42.908419+0000","last_unstale":"2023-11-08T08:18:44.271984+0000","last_undegraded":"2023-11-08T08:18:44.271984+0000","last_fullsized":"2023-11-08T08:18:44.271984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894809+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:18:42.894809+0000","last_peered":"2023-11-08T08:18:42.894809+0000","last_clean":"2023-11-08T08:18:42.894809+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:18:42.894809+0000","last_undegraded":"2023-11-08T08:18:42.894809+0000","last_fullsized":"2023-11-08T08:18:42.894809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272802+0000","last_change":"2023-11-08T08:18:42.496201+0000","last_active":"2023-11-08T08:18:44.272802+0000","last_peered":"2023-11-08T08:18:44.272802+0000","last_clean":"2023-11-08T08:18:44.272802+0000","last_became_active":"2023-11-08T08:18:42.495914+0000","last_became_peered":"2023-11-08T08:18:42.495914+0000","last_unstale":"2023-11-08T08:18:44.272802+0000","last_undegraded":"2023-11-08T08:18:44.272802+0000","last_fullsized":"2023-11-08T08:18:44.272802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893727+0000","last_change":"2023-11-08T08:18:42.907389+0000","last_active":"2023-11-08T08:18:43.893727+0000","last_peered":"2023-11-08T08:18:43.893727+0000","last_clean":"2023-11-08T08:18:43.893727+0000","last_became_active":"2023-11-08T08:18:42.906767+0000","last_became_peered":"2023-11-08T08:18:42.906767+0000","last_unstale":"2023-11-08T08:18:43.893727+0000","last_undegraded":"2023-11-08T08:18:43.893727+0000","last_fullsized":"2023-11-08T08:18:43.893727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.890889+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:18:42.890889+0000","last_peered":"2023-11-08T08:18:42.890889+0000","last_clean":"2023-11-08T08:18:42.890889+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:18:42.890889+0000","last_undegraded":"2023-11-08T08:18:42.890889+0000","last_fullsized":"2023-11-08T08:18:42.890889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.895954+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:18:42.895954+0000","last_peered":"2023-11-08T08:18:42.895954+0000","last_clean":"2023-11-08T08:18:42.895954+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:18:42.895954+0000","last_undegraded":"2023-11-08T08:18:42.895954+0000","last_fullsized":"2023-11-08T08:18:42.895954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272359+0000","last_change":"2023-11-08T08:18:42.505906+0000","last_active":"2023-11-08T08:18:44.272359+0000","last_peered":"2023-11-08T08:18:44.272359+0000","last_clean":"2023-11-08T08:18:44.272359+0000","last_became_active":"2023-11-08T08:18:42.505688+0000","last_became_peered":"2023-11-08T08:18:42.505688+0000","last_unstale":"2023-11-08T08:18:44.272359+0000","last_undegraded":"2023-11-08T08:18:44.272359+0000","last_fullsized":"2023-11-08T08:18:44.272359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899202+0000","last_change":"2023-11-08T08:18:42.907668+0000","last_active":"2023-11-08T08:18:43.899202+0000","last_peered":"2023-11-08T08:18:43.899202+0000","last_clean":"2023-11-08T08:18:43.899202+0000","last_became_active":"2023-11-08T08:18:42.907352+0000","last_became_peered":"2023-11-08T08:18:42.907352+0000","last_unstale":"2023-11-08T08:18:43.899202+0000","last_undegraded":"2023-11-08T08:18:43.899202+0000","last_fullsized":"2023-11-08T08:18:43.899202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894118+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:18:42.894118+0000","last_peered":"2023-11-08T08:18:42.894118+0000","last_clean":"2023-11-08T08:18:42.894118+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:18:42.894118+0000","last_undegraded":"2023-11-08T08:18:42.894118+0000","last_fullsized":"2023-11-08T08:18:42.894118+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898738+0000","last_change":"2023-11-08T08:18:42.491954+0000","last_active":"2023-11-08T08:18:43.898738+0000","last_peered":"2023-11-08T08:18:43.898738+0000","last_clean":"2023-11-08T08:18:43.898738+0000","last_became_active":"2023-11-08T08:18:42.491305+0000","last_became_peered":"2023-11-08T08:18:42.491305+0000","last_unstale":"2023-11-08T08:18:43.898738+0000","last_undegraded":"2023-11-08T08:18:43.898738+0000","last_fullsized":"2023-11-08T08:18:43.898738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272932+0000","last_change":"2023-11-08T08:18:42.910300+0000","last_active":"2023-11-08T08:18:44.272932+0000","last_peered":"2023-11-08T08:18:44.272932+0000","last_clean":"2023-11-08T08:18:44.272932+0000","last_became_active":"2023-11-08T08:18:42.910084+0000","last_became_peered":"2023-11-08T08:18:42.910084+0000","last_unstale":"2023-11-08T08:18:44.272932+0000","last_undegraded":"2023-11-08T08:18:44.272932+0000","last_fullsized":"2023-11-08T08:18:44.272932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.889516+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:18:42.889516+0000","last_peered":"2023-11-08T08:18:42.889516+0000","last_clean":"2023-11-08T08:18:42.889516+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:18:42.889516+0000","last_undegraded":"2023-11-08T08:18:42.889516+0000","last_fullsized":"2023-11-08T08:18:42.889516+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272717+0000","last_change":"2023-11-08T08:18:42.493859+0000","last_active":"2023-11-08T08:18:44.272717+0000","last_peered":"2023-11-08T08:18:44.272717+0000","last_clean":"2023-11-08T08:18:44.272717+0000","last_became_active":"2023-11-08T08:18:42.492594+0000","last_became_peered":"2023-11-08T08:18:42.492594+0000","last_unstale":"2023-11-08T08:18:44.272717+0000","last_undegraded":"2023-11-08T08:18:44.272717+0000","last_fullsized":"2023-11-08T08:18:44.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272644+0000","last_change":"2023-11-08T08:18:42.903762+0000","last_active":"2023-11-08T08:18:44.272644+0000","last_peered":"2023-11-08T08:18:44.272644+0000","last_clean":"2023-11-08T08:18:44.272644+0000","last_became_active":"2023-11-08T08:18:42.903182+0000","last_became_peered":"2023-11-08T08:18:42.903182+0000","last_unstale":"2023-11-08T08:18:44.272644+0000","last_undegraded":"2023-11-08T08:18:44.272644+0000","last_fullsized":"2023-11-08T08:18:44.272644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:23:54.970595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896906+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:18:42.896906+0000","last_peered":"2023-11-08T08:18:42.896906+0000","last_clean":"2023-11-08T08:18:42.896906+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:18:42.896906+0000","last_undegraded":"2023-11-08T08:18:42.896906+0000","last_fullsized":"2023-11-08T08:18:42.896906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272296+0000","last_change":"2023-11-08T08:18:42.506245+0000","last_active":"2023-11-08T08:18:44.272296+0000","last_peered":"2023-11-08T08:18:44.272296+0000","last_clean":"2023-11-08T08:18:44.272296+0000","last_became_active":"2023-11-08T08:18:42.505955+0000","last_became_peered":"2023-11-08T08:18:42.505955+0000","last_unstale":"2023-11-08T08:18:44.272296+0000","last_undegraded":"2023-11-08T08:18:44.272296+0000","last_fullsized":"2023-11-08T08:18:44.272296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272936+0000","last_change":"2023-11-08T08:18:42.905213+0000","last_active":"2023-11-08T08:18:44.272936+0000","last_peered":"2023-11-08T08:18:44.272936+0000","last_clean":"2023-11-08T08:18:44.272936+0000","last_became_active":"2023-11-08T08:18:42.904949+0000","last_became_peered":"2023-11-08T08:18:42.904949+0000","last_unstale":"2023-11-08T08:18:44.272936+0000","last_undegraded":"2023-11-08T08:18:44.272936+0000","last_fullsized":"2023-11-08T08:18:44.272936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896829+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:18:42.896829+0000","last_peered":"2023-11-08T08:18:42.896829+0000","last_clean":"2023-11-08T08:18:42.896829+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:18:42.896829+0000","last_undegraded":"2023-11-08T08:18:42.896829+0000","last_fullsized":"2023-11-08T08:18:42.896829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272802+0000","last_change":"2023-11-08T08:18:42.502871+0000","last_active":"2023-11-08T08:18:44.272802+0000","last_peered":"2023-11-08T08:18:44.272802+0000","last_clean":"2023-11-08T08:18:44.272802+0000","last_became_active":"2023-11-08T08:18:42.502339+0000","last_became_peered":"2023-11-08T08:18:42.502339+0000","last_unstale":"2023-11-08T08:18:44.272802+0000","last_undegraded":"2023-11-08T08:18:44.272802+0000","last_fullsized":"2023-11-08T08:18:44.272802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:11:07.923861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272804+0000","last_change":"2023-11-08T08:18:42.910283+0000","last_active":"2023-11-08T08:18:44.272804+0000","last_peered":"2023-11-08T08:18:44.272804+0000","last_clean":"2023-11-08T08:18:44.272804+0000","last_became_active":"2023-11-08T08:18:42.909986+0000","last_became_peered":"2023-11-08T08:18:42.909986+0000","last_unstale":"2023-11-08T08:18:44.272804+0000","last_undegraded":"2023-11-08T08:18:44.272804+0000","last_fullsized":"2023-11-08T08:18:44.272804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.892038+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:18:42.892038+0000","last_peered":"2023-11-08T08:18:42.892038+0000","last_clean":"2023-11-08T08:18:42.892038+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:18:42.892038+0000","last_undegraded":"2023-11-08T08:18:42.892038+0000","last_fullsized":"2023-11-08T08:18:42.892038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893889+0000","last_change":"2023-11-08T08:18:42.503455+0000","last_active":"2023-11-08T08:18:43.893889+0000","last_peered":"2023-11-08T08:18:43.893889+0000","last_clean":"2023-11-08T08:18:43.893889+0000","last_became_active":"2023-11-08T08:18:42.503187+0000","last_became_peered":"2023-11-08T08:18:42.503187+0000","last_unstale":"2023-11-08T08:18:43.893889+0000","last_undegraded":"2023-11-08T08:18:43.893889+0000","last_fullsized":"2023-11-08T08:18:43.893889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272626+0000","last_change":"2023-11-08T08:18:42.911640+0000","last_active":"2023-11-08T08:18:44.272626+0000","last_peered":"2023-11-08T08:18:44.272626+0000","last_clean":"2023-11-08T08:18:44.272626+0000","last_became_active":"2023-11-08T08:18:42.911415+0000","last_became_peered":"2023-11-08T08:18:42.911415+0000","last_unstale":"2023-11-08T08:18:44.272626+0000","last_undegraded":"2023-11-08T08:18:44.272626+0000","last_fullsized":"2023-11-08T08:18:44.272626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272344+0000","last_change":"2023-11-08T08:18:42.501133+0000","last_active":"2023-11-08T08:18:44.272344+0000","last_peered":"2023-11-08T08:18:44.272344+0000","last_clean":"2023-11-08T08:18:44.272344+0000","last_became_active":"2023-11-08T08:18:42.500842+0000","last_became_peered":"2023-11-08T08:18:42.500842+0000","last_unstale":"2023-11-08T08:18:44.272344+0000","last_undegraded":"2023-11-08T08:18:44.272344+0000","last_fullsized":"2023-11-08T08:18:44.272344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:05.271102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272942+0000","last_change":"2023-11-08T08:18:42.905081+0000","last_active":"2023-11-08T08:18:44.272942+0000","last_peered":"2023-11-08T08:18:44.272942+0000","last_clean":"2023-11-08T08:18:44.272942+0000","last_became_active":"2023-11-08T08:18:42.904781+0000","last_became_peered":"2023-11-08T08:18:42.904781+0000","last_unstale":"2023-11-08T08:18:44.272942+0000","last_undegraded":"2023-11-08T08:18:44.272942+0000","last_fullsized":"2023-11-08T08:18:44.272942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898380+0000","last_change":"2023-11-08T08:18:42.505921+0000","last_active":"2023-11-08T08:18:43.898380+0000","last_peered":"2023-11-08T08:18:43.898380+0000","last_clean":"2023-11-08T08:18:43.898380+0000","last_became_active":"2023-11-08T08:18:42.505591+0000","last_became_peered":"2023-11-08T08:18:42.505591+0000","last_unstale":"2023-11-08T08:18:43.898380+0000","last_undegraded":"2023-11-08T08:18:43.898380+0000","last_fullsized":"2023-11-08T08:18:43.898380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893978+0000","last_change":"2023-11-08T08:18:42.907871+0000","last_active":"2023-11-08T08:18:43.893978+0000","last_peered":"2023-11-08T08:18:43.893978+0000","last_clean":"2023-11-08T08:18:43.893978+0000","last_became_active":"2023-11-08T08:18:42.907170+0000","last_became_peered":"2023-11-08T08:18:42.907170+0000","last_unstale":"2023-11-08T08:18:43.893978+0000","last_undegraded":"2023-11-08T08:18:43.893978+0000","last_fullsized":"2023-11-08T08:18:43.893978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272471+0000","last_change":"2023-11-08T08:18:42.504796+0000","last_active":"2023-11-08T08:18:44.272471+0000","last_peered":"2023-11-08T08:18:44.272471+0000","last_clean":"2023-11-08T08:18:44.272471+0000","last_became_active":"2023-11-08T08:18:42.504498+0000","last_became_peered":"2023-11-08T08:18:42.504498+0000","last_unstale":"2023-11-08T08:18:44.272471+0000","last_undegraded":"2023-11-08T08:18:44.272471+0000","last_fullsized":"2023-11-08T08:18:44.272471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:02.270931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893958+0000","last_change":"2023-11-08T08:18:42.906660+0000","last_active":"2023-11-08T08:18:43.893958+0000","last_peered":"2023-11-08T08:18:43.893958+0000","last_clean":"2023-11-08T08:18:43.893958+0000","last_became_active":"2023-11-08T08:18:42.906410+0000","last_became_peered":"2023-11-08T08:18:42.906410+0000","last_unstale":"2023-11-08T08:18:43.893958+0000","last_undegraded":"2023-11-08T08:18:43.893958+0000","last_fullsized":"2023-11-08T08:18:43.893958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899130+0000","last_change":"2023-11-08T08:18:42.497356+0000","last_active":"2023-11-08T08:18:43.899130+0000","last_peered":"2023-11-08T08:18:43.899130+0000","last_clean":"2023-11-08T08:18:43.899130+0000","last_became_active":"2023-11-08T08:18:42.497056+0000","last_became_peered":"2023-11-08T08:18:42.497056+0000","last_unstale":"2023-11-08T08:18:43.899130+0000","last_undegraded":"2023-11-08T08:18:43.899130+0000","last_fullsized":"2023-11-08T08:18:43.899130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272096+0000","last_change":"2023-11-08T08:18:42.905346+0000","last_active":"2023-11-08T08:18:44.272096+0000","last_peered":"2023-11-08T08:18:44.272096+0000","last_clean":"2023-11-08T08:18:44.272096+0000","last_became_active":"2023-11-08T08:18:42.905050+0000","last_became_peered":"2023-11-08T08:18:42.905050+0000","last_unstale":"2023-11-08T08:18:44.272096+0000","last_undegraded":"2023-11-08T08:18:44.272096+0000","last_fullsized":"2023-11-08T08:18:44.272096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899328+0000","last_change":"2023-11-08T08:18:42.499349+0000","last_active":"2023-11-08T08:18:43.899328+0000","last_peered":"2023-11-08T08:18:43.899328+0000","last_clean":"2023-11-08T08:18:43.899328+0000","last_became_active":"2023-11-08T08:18:42.499050+0000","last_became_peered":"2023-11-08T08:18:42.499050+0000","last_unstale":"2023-11-08T08:18:43.899328+0000","last_undegraded":"2023-11-08T08:18:43.899328+0000","last_fullsized":"2023-11-08T08:18:43.899328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893668+0000","last_change":"2023-11-08T08:18:42.913391+0000","last_active":"2023-11-08T08:18:43.893668+0000","last_peered":"2023-11-08T08:18:43.893668+0000","last_clean":"2023-11-08T08:18:43.893668+0000","last_became_active":"2023-11-08T08:18:42.913124+0000","last_became_peered":"2023-11-08T08:18:42.913124+0000","last_unstale":"2023-11-08T08:18:43.893668+0000","last_undegraded":"2023-11-08T08:18:43.893668+0000","last_fullsized":"2023-11-08T08:18:43.893668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898165+0000","last_change":"2023-11-08T08:18:42.912208+0000","last_active":"2023-11-08T08:18:43.898165+0000","last_peered":"2023-11-08T08:18:43.898165+0000","last_clean":"2023-11-08T08:18:43.898165+0000","last_became_active":"2023-11-08T08:18:42.911928+0000","last_became_peered":"2023-11-08T08:18:42.911928+0000","last_unstale":"2023-11-08T08:18:43.898165+0000","last_undegraded":"2023-11-08T08:18:43.898165+0000","last_fullsized":"2023-11-08T08:18:43.898165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272255+0000","last_change":"2023-11-08T08:18:42.914191+0000","last_active":"2023-11-08T08:18:44.272255+0000","last_peered":"2023-11-08T08:18:44.272255+0000","last_clean":"2023-11-08T08:18:44.272255+0000","last_became_active":"2023-11-08T08:18:42.913919+0000","last_became_peered":"2023-11-08T08:18:42.913919+0000","last_unstale":"2023-11-08T08:18:44.272255+0000","last_undegraded":"2023-11-08T08:18:44.272255+0000","last_fullsized":"2023-11-08T08:18:44.272255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271708+0000","last_change":"2023-11-08T08:18:42.900802+0000","last_active":"2023-11-08T08:18:44.271708+0000","last_peered":"2023-11-08T08:18:44.271708+0000","last_clean":"2023-11-08T08:18:44.271708+0000","last_became_active":"2023-11-08T08:18:42.900538+0000","last_became_peered":"2023-11-08T08:18:42.900538+0000","last_unstale":"2023-11-08T08:18:44.271708+0000","last_undegraded":"2023-11-08T08:18:44.271708+0000","last_fullsized":"2023-11-08T08:18:44.271708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899133+0000","last_change":"2023-11-08T08:18:42.906606+0000","last_active":"2023-11-08T08:18:43.899133+0000","last_peered":"2023-11-08T08:18:43.899133+0000","last_clean":"2023-11-08T08:18:43.899133+0000","last_became_active":"2023-11-08T08:18:42.906279+0000","last_became_peered":"2023-11-08T08:18:42.906279+0000","last_unstale":"2023-11-08T08:18:43.899133+0000","last_undegraded":"2023-11-08T08:18:43.899133+0000","last_fullsized":"2023-11-08T08:18:43.899133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899035+0000","last_change":"2023-11-08T08:18:42.915689+0000","last_active":"2023-11-08T08:18:43.899035+0000","last_peered":"2023-11-08T08:18:43.899035+0000","last_clean":"2023-11-08T08:18:43.899035+0000","last_became_active":"2023-11-08T08:18:42.915121+0000","last_became_peered":"2023-11-08T08:18:42.915121+0000","last_unstale":"2023-11-08T08:18:43.899035+0000","last_undegraded":"2023-11-08T08:18:43.899035+0000","last_fullsized":"2023-11-08T08:18:43.899035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7888,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363952,"statfs":{"total":96636764160,"available":96628686848,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":73265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":74582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2759999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9064,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362776,"statfs":{"total":96636764160,"available":96627482624,"internally_reserved":0,"allocated":819200,"data_stored":666456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9048,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362792,"statfs":{"total":96636764160,"available":96627499008,"internally_reserved":0,"allocated":802816,"data_stored":665370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:18:47.324 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-08T08:18:47.324 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-08T08:18:47.325 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-08T08:18:47.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-08T08:18:47.326 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-08T08:18:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.478+0000 7ffac0d7d700 1 Processor -- start 2023-11-08T08:18:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.478+0000 7ffac0d7d700 1 -- start start 2023-11-08T08:18:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.479+0000 7ffac0d7d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc06a270 0x7ffabc071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.479+0000 7ffac0d7d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffabc0ac420 0x7ffabc0ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.479+0000 7ffac0d7d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0af0c0 con 0x7ffabc0ae520 2023-11-08T08:18:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.479+0000 7ffac0d7d700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffabc0b0590 con 0x7ffabc0ac420 2023-11-08T08:18:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.479+0000 7ffac0d7d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc159280 con 0x7ffabc06a270 2023-11-08T08:18:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.479+0000 7ffabad9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffabc0ac420 0x7ffabc0ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffabc0ac420 0x7ffabc0ac8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39098/0 (socket says 172.21.15.116:39098) 2023-11-08T08:18:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 -- 172.21.15.116:0/3716079555 learned_addr learned my addr 172.21.15.116:0/3716079555 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffaba59c700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc06a270 0x7ffabc071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 -- 172.21.15.116:0/3716079555 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc06a270 msgr2=0x7ffabc071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:47.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc06a270 0x7ffabc071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 -- 172.21.15.116:0/3716079555 >> v1:172.21.15.129:6789/0 conn(0x7ffabc0ae520 legacy=0x7ffabc06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 -- 172.21.15.116:0/3716079555 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0b0d10 con 0x7ffabc0ac420 2023-11-08T08:18:47.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffaba59c700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc06a270 0x7ffabc071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:47.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffabad9d700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc0ac420 0x7ffabc0ac8a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffab400d150 tx=0x7ffab400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66abcad396748d32 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffab959a700 1 -- 172.21.15.116:0/3716079555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4010070 con 0x7ffabc0ac420 2023-11-08T08:18:47.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffab959a700 1 -- 172.21.15.116:0/3716079555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab400dbb0 con 0x7ffabc0ac420 2023-11-08T08:18:47.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.480+0000 7ffab959a700 1 -- 172.21.15.116:0/3716079555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4016400 con 0x7ffabc0ac420 2023-11-08T08:18:47.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 -- 172.21.15.116:0/3716079555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc0ac420 msgr2=0x7ffabc0ac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc0ac420 0x7ffabc0ac8a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffab400d150 tx=0x7ffab400d460 comp rx=0 tx=0).stop 2023-11-08T08:18:47.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 -- 172.21.15.116:0/3716079555 shutdown_connections 2023-11-08T08:18:47.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc0ac420 0x7ffabc0ac8a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/3716079555 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc06a270 0x7ffabc071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 -- 172.21.15.116:0/3716079555 >> 172.21.15.116:0/3716079555 conn(0x7ffabc0b2fc0 msgr2=0x7ffabc071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 -- 172.21.15.116:0/3716079555 shutdown_connections 2023-11-08T08:18:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.481+0000 7ffac0d7d700 1 -- 172.21.15.116:0/3716079555 wait complete. 2023-11-08T08:18:47.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.482+0000 7ffac0d7d700 1 Processor -- start 2023-11-08T08:18:47.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.482+0000 7ffac0d7d700 1 -- start start 2023-11-08T08:18:47.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 0x7ffabc0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc0ac420 0x7ffabc1e2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc0ae520 0x7ffabc1e2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc1e2bc0 con 0x7ffabc06a270 2023-11-08T08:18:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc1e2ea0 con 0x7ffabc0ae520 2023-11-08T08:18:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc1e3180 con 0x7ffabc0ac420 2023-11-08T08:18:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 0x7ffabc0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 0x7ffabc0cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39106/0 (socket says 172.21.15.116:39106) 2023-11-08T08:18:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 -- 172.21.15.116:0/2558371865 learned_addr learned my addr 172.21.15.116:0/2558371865 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffab9d9b700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc0ac420 0x7ffabc1e2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffabad9d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc0ae520 0x7ffabc1e2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 -- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc0ac420 msgr2=0x7ffabc1e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc0ac420 0x7ffabc1e2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 -- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc0ae520 msgr2=0x7ffabc1e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc0ae520 0x7ffabc1e2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 -- 172.21.15.116:0/2558371865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab400bd40 con 0x7ffabc06a270 2023-11-08T08:18:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaba59c700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 0x7ffabc0cbd60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e760 tx=0x7ffab000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0013610 con 0x7ffabc06a270 2023-11-08T08:18:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab0013770 con 0x7ffabc06a270 2023-11-08T08:18:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc0afc60 con 0x7ffabc06a270 2023-11-08T08:18:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0013b20 con 0x7ffabc06a270 2023-11-08T08:18:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.483+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc072670 con 0x7ffabc06a270 2023-11-08T08:18:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.484+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffabc0aff40 con 0x7ffabc06a270 2023-11-08T08:18:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.485+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffab0023070 con 0x7ffabc06a270 2023-11-08T08:18:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.485+0000 7ffaaaffd700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffa9c0682f0 0x7ffa9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.485+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ffab0085830 con 0x7ffabc06a270 2023-11-08T08:18:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.485+0000 7ffab9d9b700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffa9c0682f0 0x7ffa9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.486+0000 7ffab9d9b700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffa9c0682f0 0x7ffa9c06a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0cb870 tx=0x7ffaac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.489+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffab0052090 con 0x7ffabc06a270 2023-11-08T08:18:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.719+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ffabc0743f0 con 0x7ffabc06a270 2023-11-08T08:18:47.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.722+0000 7ffaaaffd700 1 -- 172.21.15.116:0/2558371865 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+166386 (secure 0 0 0) 0x7ffab00556b0 con 0x7ffabc06a270 2023-11-08T08:18:47.724 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:47.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffa9c0682f0 msgr2=0x7ffa9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffa9c0682f0 0x7ffa9c06a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0cb870 tx=0x7ffaac009040 comp rx=0 tx=0).stop 2023-11-08T08:18:47.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 msgr2=0x7ffabc0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 0x7ffabc0cbd60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e760 tx=0x7ffab000eb20 comp rx=0 tx=0).stop 2023-11-08T08:18:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 shutdown_connections 2023-11-08T08:18:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffa9c0682f0 0x7ffa9c06a7a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc0ae520 0x7ffabc1e2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffabc06a270 0x7ffabc0cbd60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 --2- 172.21.15.116:0/2558371865 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffabc0ac420 0x7ffabc1e2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 >> 172.21.15.116:0/2558371865 conn(0x7ffabc0b2fc0 msgr2=0x7ffabc06c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:47.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 shutdown_connections 2023-11-08T08:18:47.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.725+0000 7ffac0d7d700 1 -- 172.21.15.116:0/2558371865 wait complete. 2023-11-08T08:18:47.736 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":120856},{"type":"v1","addr":"172.21.15.116:6833","nonce":120856}]},"active_addr":"172.21.15.116:6833/120856","active_change":"2023-11-08T08:18:30.345341+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2766130363}]},{"addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2632295104}]},{"addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1319718174}]},{"addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3212645320}]}]} 2023-11-08T08:18:47.741 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-08T08:18:47.741 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-08T08:18:47.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.860+0000 7f14bb478700 1 Processor -- start 2023-11-08T08:18:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.860+0000 7f14bb478700 1 -- start start 2023-11-08T08:18:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14bb478700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ced70 0x7f14b40b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14bb478700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14b40ce1e0 0x7f14b40b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14bb478700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b40b4fc0 con 0x7f14b406feb0 2023-11-08T08:18:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14bb478700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b4159c30 con 0x7f14b40ce1e0 2023-11-08T08:18:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14bb478700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40b05e0 con 0x7f14b40ced70 2023-11-08T08:18:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9a15700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14b40ce1e0 0x7f14b40b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9a15700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14b40ce1e0 0x7f14b40b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39110/0 (socket says 172.21.15.116:39110) 2023-11-08T08:18:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9a15700 1 -- 172.21.15.116:0/2546271078 learned_addr learned my addr 172.21.15.116:0/2546271078 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9214700 1 --2- 172.21.15.116:0/2546271078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ced70 0x7f14b40b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9a15700 1 -- 172.21.15.116:0/2546271078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ced70 msgr2=0x7f14b40b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9a15700 1 --2- 172.21.15.116:0/2546271078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ced70 0x7f14b40b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.861+0000 7f14b9a15700 1 -- 172.21.15.116:0/2546271078 >> v1:172.21.15.129:6789/0 conn(0x7f14b406feb0 legacy=0x7f14b406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14b9a15700 1 -- 172.21.15.116:0/2546271078 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40b08c0 con 0x7f14b40ce1e0 2023-11-08T08:18:47.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14b9a15700 1 --2- 172.21.15.116:0/2546271078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ce1e0 0x7f14b40b4930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009e50 tx=0x7f14ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfa5a0bfc728ad8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14abfff700 1 -- 172.21.15.116:0/2546271078 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac00e070 con 0x7f14b40ce1e0 2023-11-08T08:18:47.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14abfff700 1 -- 172.21.15.116:0/2546271078 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ac00b9c0 con 0x7f14b40ce1e0 2023-11-08T08:18:47.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14abfff700 1 -- 172.21.15.116:0/2546271078 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac00bd70 con 0x7f14b40ce1e0 2023-11-08T08:18:47.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 -- 172.21.15.116:0/2546271078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ce1e0 msgr2=0x7f14b40b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 --2- 172.21.15.116:0/2546271078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ce1e0 0x7f14b40b4930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009e50 tx=0x7f14ac00b330 comp rx=0 tx=0).stop 2023-11-08T08:18:47.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 -- 172.21.15.116:0/2546271078 shutdown_connections 2023-11-08T08:18:47.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 --2- 172.21.15.116:0/2546271078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ce1e0 0x7f14b40b4930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 --2- 172.21.15.116:0/2546271078 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ced70 0x7f14b40b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 -- 172.21.15.116:0/2546271078 >> 172.21.15.116:0/2546271078 conn(0x7f14b40b2fc0 msgr2=0x7f14b4071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 -- 172.21.15.116:0/2546271078 shutdown_connections 2023-11-08T08:18:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.862+0000 7f14bb478700 1 -- 172.21.15.116:0/2546271078 wait complete. 2023-11-08T08:18:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 Processor -- start 2023-11-08T08:18:47.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 -- start start 2023-11-08T08:18:47.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14b406feb0 0x7f14b41efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ce1e0 0x7f14b40c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 0x7f14b40c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b415baf0 con 0x7f14b40ced70 2023-11-08T08:18:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b415bdd0 con 0x7f14b406feb0 2023-11-08T08:18:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14bb478700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b415c0b0 con 0x7f14b40ce1e0 2023-11-08T08:18:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14b9a15700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 0x7f14b40c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14b9a15700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 0x7f14b40c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39120/0 (socket says 172.21.15.116:39120) 2023-11-08T08:18:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14b9a15700 1 -- 172.21.15.116:0/4260459906 learned_addr learned my addr 172.21.15.116:0/4260459906 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14b9214700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14b406feb0 0x7f14b41efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.864+0000 7f14b8a13700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ce1e0 0x7f14b40c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14b9a15700 1 -- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ce1e0 msgr2=0x7f14b40c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14b9a15700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ce1e0 0x7f14b40c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14b9a15700 1 -- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14b406feb0 msgr2=0x7f14b41efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14b9a15700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14b406feb0 0x7f14b41efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14b9a15700 1 -- 172.21.15.116:0/4260459906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac009b30 con 0x7f14b40ced70 2023-11-08T08:18:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14b9a15700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 0x7f14b40c53b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f14b4073cc0 tx=0x7f14ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac004800 con 0x7f14b40ced70 2023-11-08T08:18:47.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ac004960 con 0x7f14b40ced70 2023-11-08T08:18:47.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b40b5260 con 0x7f14b40ced70 2023-11-08T08:18:47.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac01c680 con 0x7f14b40ced70 2023-11-08T08:18:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.865+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b40b0ba0 con 0x7f14b40ced70 2023-11-08T08:18:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.866+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b4072420 con 0x7f14b40ced70 2023-11-08T08:18:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.866+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f14ac022030 con 0x7f14b40ced70 2023-11-08T08:18:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.867+0000 7f14a9ffb700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14980682f0 0x7f149806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:47.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.867+0000 7f14b9214700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14980682f0 0x7f149806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:47.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.867+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f14ac087340 con 0x7f14b40ced70 2023-11-08T08:18:47.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.867+0000 7f14b9214700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14980682f0 0x7f149806a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006b20 tx=0x7f14a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:47.871+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f14ac053ba0 con 0x7f14b40ced70 2023-11-08T08:18:48.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.054+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14b40661b0 con 0x7f14b40ced70 2023-11-08T08:18:48.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.055+0000 7f14a9ffb700 1 -- 172.21.15.116:0/4260459906 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f14ac0571c0 con 0x7f14b40ced70 2023-11-08T08:18:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:48.056 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:48.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14980682f0 msgr2=0x7f149806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14980682f0 0x7f149806a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006b20 tx=0x7f14a4008040 comp rx=0 tx=0).stop 2023-11-08T08:18:48.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 msgr2=0x7f14b40c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 0x7f14b40c53b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f14b4073cc0 tx=0x7f14ac004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 shutdown_connections 2023-11-08T08:18:48.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b40ced70 0x7f14b40c53b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14980682f0 0x7f149806a7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14b406feb0 0x7f14b41efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 --2- 172.21.15.116:0/4260459906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14b40ce1e0 0x7f14b40c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 >> 172.21.15.116:0/4260459906 conn(0x7f14b40b2fc0 msgr2=0x7f14b4159350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 shutdown_connections 2023-11-08T08:18:48.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.057+0000 7f14bb478700 1 -- 172.21.15.116:0/4260459906 wait complete. 2023-11-08T08:18:48.069 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-08T08:18:48.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 Processor -- start 2023-11-08T08:18:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 -- start start 2023-11-08T08:18:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f61940b0880 0x7f619406b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f61940ad1f0 0x7f61940ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61940adb50 con 0x7f6194072180 2023-11-08T08:18:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6194072620 con 0x7f61940ad1f0 2023-11-08T08:18:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.189+0000 7f6199f17700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61940b43f0 con 0x7f61940b0880 2023-11-08T08:18:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6193fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f61940ad1f0 0x7f61940ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6192ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6194072180 0x7f61940aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34280/0 (socket says 172.21.15.116:34280) 2023-11-08T08:18:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6192ffd700 1 -- 172.21.15.116:0/723824807 learned_addr learned my addr 172.21.15.116:0/723824807 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6193fff700 1 -- 172.21.15.116:0/723824807 >> v2:172.21.15.129:3300/0 conn(0x7f61940b0880 msgr2=0x7f619406b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6193fff700 1 --2- 172.21.15.116:0/723824807 >> v2:172.21.15.129:3300/0 conn(0x7f61940b0880 0x7f619406b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6193fff700 1 -- 172.21.15.116:0/723824807 >> v1:172.21.15.116:6789/0 conn(0x7f6194072180 legacy=0x7f61940aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f61927fc700 1 -- 172.21.15.116:0/723824807 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142442234 0 0) 0x7f61940adb50 con 0x7f6194072180 2023-11-08T08:18:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6193fff700 1 -- 172.21.15.116:0/723824807 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61940b46d0 con 0x7f61940ad1f0 2023-11-08T08:18:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f6193fff700 1 --2- 172.21.15.116:0/723824807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f61940ad610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f619406efd0 tx=0x7f618c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7944bbe73ce3f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f61927fc700 1 -- 172.21.15.116:0/723824807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c00e070 con 0x7f61940ad1f0 2023-11-08T08:18:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f61927fc700 1 -- 172.21.15.116:0/723824807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c00bbd0 con 0x7f61940ad1f0 2023-11-08T08:18:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.190+0000 7f61927fc700 1 -- 172.21.15.116:0/723824807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c012430 con 0x7f61940ad1f0 2023-11-08T08:18:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 -- 172.21.15.116:0/723824807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 msgr2=0x7f61940ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 --2- 172.21.15.116:0/723824807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f61940ad610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f619406efd0 tx=0x7f618c00b330 comp rx=0 tx=0).stop 2023-11-08T08:18:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 -- 172.21.15.116:0/723824807 shutdown_connections 2023-11-08T08:18:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 --2- 172.21.15.116:0/723824807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f61940ad610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 --2- 172.21.15.116:0/723824807 >> v2:172.21.15.129:3300/0 conn(0x7f61940b0880 0x7f619406b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 -- 172.21.15.116:0/723824807 >> 172.21.15.116:0/723824807 conn(0x7f61940b2fc0 msgr2=0x7f6194071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 -- 172.21.15.116:0/723824807 shutdown_connections 2023-11-08T08:18:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.191+0000 7f6199f17700 1 -- 172.21.15.116:0/723824807 wait complete. 2023-11-08T08:18:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.192+0000 7f6199f17700 1 Processor -- start 2023-11-08T08:18:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.192+0000 7f6199f17700 1 -- start start 2023-11-08T08:18:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6199f17700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6194072180 0x7f6194098a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6199f17700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f619406ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6199f17700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61940b0880 0x7f619406cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6199f17700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619409bcf0 con 0x7f61940ad1f0 2023-11-08T08:18:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6199f17700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6194098f70 con 0x7f61940b0880 2023-11-08T08:18:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6199f17700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6194099490 con 0x7f6194072180 2023-11-08T08:18:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6192ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f619406ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6192ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f619406ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39126/0 (socket says 172.21.15.116:39126) 2023-11-08T08:18:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6192ffd700 1 -- 172.21.15.116:0/2082139574 learned_addr learned my addr 172.21.15.116:0/2082139574 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.193+0000 7f6193fff700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61940b0880 0x7f619406cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f61937fe700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6194072180 0x7f6194098a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6192ffd700 1 -- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6194072180 msgr2=0x7f6194098a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6192ffd700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6194072180 0x7f6194098a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6192ffd700 1 -- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61940b0880 msgr2=0x7f619406cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6192ffd700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61940b0880 0x7f619406cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6192ffd700 1 -- 172.21.15.116:0/2082139574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c009a70 con 0x7f61940ad1f0 2023-11-08T08:18:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6193fff700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61940b0880 0x7f619406cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6192ffd700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f619406ca90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f618400c550 tx=0x7f618400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184004c80 con 0x7f61940ad1f0 2023-11-08T08:18:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184004e00 con 0x7f61940ad1f0 2023-11-08T08:18:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184007310 con 0x7f61940ad1f0 2023-11-08T08:18:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61940b4fe0 con 0x7f61940ad1f0 2023-11-08T08:18:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.194+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61940b4d00 con 0x7f61940ad1f0 2023-11-08T08:18:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.195+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61940b49b0 con 0x7f61940ad1f0 2023-11-08T08:18:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.195+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f618401a020 con 0x7f61940ad1f0 2023-11-08T08:18:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.195+0000 7f618bfff700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f61740682f0 0x7f617406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.196+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6184085840 con 0x7f61940ad1f0 2023-11-08T08:18:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.196+0000 7f61937fe700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f61740682f0 0x7f617406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.196+0000 7f61937fe700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f61740682f0 0x7f617406a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f617c006fd0 tx=0x7f617c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.200+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6184052150 con 0x7f61940ad1f0 2023-11-08T08:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.380+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61940743f0 con 0x7f61940ad1f0 2023-11-08T08:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.381+0000 7f618bfff700 1 -- 172.21.15.116:0/2082139574 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f6184055770 con 0x7f61940ad1f0 2023-11-08T08:18:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:48.381 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:48.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f61740682f0 msgr2=0x7f617406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f61740682f0 0x7f617406a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f617c006fd0 tx=0x7f617c008040 comp rx=0 tx=0).stop 2023-11-08T08:18:48.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 msgr2=0x7f619406ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f619406ca90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f618400c550 tx=0x7f618400c910 comp rx=0 tx=0).stop 2023-11-08T08:18:48.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 shutdown_connections 2023-11-08T08:18:48.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f61740682f0 0x7f617406a7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61940b0880 0x7f619406cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6194072180 0x7f6194098a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 --2- 172.21.15.116:0/2082139574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61940ad1f0 0x7f619406ca90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 >> 172.21.15.116:0/2082139574 conn(0x7f61940b2fc0 msgr2=0x7f619406dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 shutdown_connections 2023-11-08T08:18:48.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.383+0000 7f6199f17700 1 -- 172.21.15.116:0/2082139574 wait complete. 2023-11-08T08:18:48.395 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-08T08:18:48.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-08T08:18:48.397 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-08T08:18:48.397 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-08T08:18:48.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-08T08:18:48.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-08T08:18:48.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-08T08:18:48.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-08T08:18:48.447 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:18:48.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:18:48.605 INFO:tasks.ceph.mds.d.smithi129.stderr:2023-11-08T08:18:48.603+0000 7f3713850700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:18:48.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.607+0000 7f63c7f46700 1 Processor -- start 2023-11-08T08:18:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.608+0000 7f63c7f46700 1 -- start start 2023-11-08T08:18:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.609+0000 7f63c7f46700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63c01609c0 0x7f63c00ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.609+0000 7f63c7f46700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f63c006cdd0 0x7f63c00b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.609+0000 7f63c7f46700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c00cc690 con 0x7f63c00b0a80 2023-11-08T08:18:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.609+0000 7f63c7f46700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c00b0ea0 con 0x7f63c01609c0 2023-11-08T08:18:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.609+0000 7f63c7f46700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c00cc8c0 con 0x7f63c006cdd0 2023-11-08T08:18:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63c01609c0 0x7f63c00ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63c01609c0 0x7f63c00ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39138/0 (socket says 172.21.15.116:39138) 2023-11-08T08:18:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 -- 172.21.15.116:0/3676883430 learned_addr learned my addr 172.21.15.116:0/3676883430 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 -- 172.21.15.116:0/3676883430 >> v2:172.21.15.129:3300/0 conn(0x7f63c006cdd0 msgr2=0x7f63c00b0540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:18:48.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 --2- 172.21.15.116:0/3676883430 >> v2:172.21.15.129:3300/0 conn(0x7f63c006cdd0 0x7f63c00b0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 -- 172.21.15.116:0/3676883430 >> v1:172.21.15.129:6789/0 conn(0x7f63c00b0a80 legacy=0x7f63c0164630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 -- 172.21.15.116:0/3676883430 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00caa30 con 0x7f63c01609c0 2023-11-08T08:18:48.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.610+0000 7f63c5ce2700 1 --2- 172.21.15.116:0/3676883430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c00ca150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f63c00672e0 tx=0x7f63b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b530860636f188 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.611+0000 7fa4f9ee8700 1 Processor -- start 2023-11-08T08:18:48.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.611+0000 7f63c4ce0700 1 -- 172.21.15.116:0/3676883430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0004230 con 0x7f63c01609c0 2023-11-08T08:18:48.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.611+0000 7f63c4ce0700 1 -- 172.21.15.116:0/3676883430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b0004390 con 0x7f63c01609c0 2023-11-08T08:18:48.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.611+0000 7f63c4ce0700 1 -- 172.21.15.116:0/3676883430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00036a0 con 0x7f63c01609c0 2023-11-08T08:18:48.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7f63c7f46700 1 -- 172.21.15.116:0/3676883430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 msgr2=0x7f63c00ca150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7f63c7f46700 1 --2- 172.21.15.116:0/3676883430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c00ca150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f63c00672e0 tx=0x7f63b0009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.611+0000 7fa4f9ee8700 1 -- start start 2023-11-08T08:18:48.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f9ee8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa4f40b8220 0x7fa4f40b6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f9ee8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4f40b5100 0x7fa4f40d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f9ee8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f40b85f0 con 0x7fa4f40b25d0 2023-11-08T08:18:48.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f9ee8700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f40d2580 con 0x7fa4f40b5100 2023-11-08T08:18:48.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f9ee8700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f40d2860 con 0x7fa4f40b8220 2023-11-08T08:18:48.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7f63c7f46700 1 -- 172.21.15.116:0/3676883430 shutdown_connections 2023-11-08T08:18:48.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7f63c7f46700 1 --2- 172.21.15.116:0/3676883430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c00ca150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7f63c7f46700 1 --2- 172.21.15.116:0/3676883430 >> v2:172.21.15.129:3300/0 conn(0x7f63c006cdd0 0x7f63c00b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7f63c7f46700 1 -- 172.21.15.116:0/3676883430 >> 172.21.15.116:0/3676883430 conn(0x7f63c006aed0 msgr2=0x7f63c00cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.613+0000 7f63c7f46700 1 -- 172.21.15.116:0/3676883430 shutdown_connections 2023-11-08T08:18:48.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.613+0000 7f63c7f46700 1 -- 172.21.15.116:0/3676883430 wait complete. 2023-11-08T08:18:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f2ffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa4f40b25d0 0x7fa4f40b49f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57260/0 (socket says 172.21.15.116:57260) 2023-11-08T08:18:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.612+0000 7fa4f2ffd700 1 -- 172.21.15.116:0/4011141353 learned_addr learned my addr 172.21.15.116:0/4011141353 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.613+0000 7fa4f37fe700 1 --2- 172.21.15.116:0/4011141353 >> v2:172.21.15.129:3300/0 conn(0x7fa4f40b8220 0x7fa4f40b6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.613+0000 7fa4f3fff700 1 --2- 172.21.15.116:0/4011141353 >> v2:172.21.15.116:3300/0 conn(0x7fa4f40b5100 0x7fa4f40d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7fa4f3fff700 1 -- 172.21.15.116:0/4011141353 >> v2:172.21.15.129:3300/0 conn(0x7fa4f40b8220 msgr2=0x7fa4f40b6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7fa4f3fff700 1 --2- 172.21.15.116:0/4011141353 >> v2:172.21.15.129:3300/0 conn(0x7fa4f40b8220 0x7fa4f40b6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7fa4f3fff700 1 -- 172.21.15.116:0/4011141353 >> v1:172.21.15.129:6789/0 conn(0x7fa4f40b25d0 legacy=0x7fa4f40b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7fa4f3fff700 1 -- 172.21.15.116:0/4011141353 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f40d2b40 con 0x7fa4f40b5100 2023-11-08T08:18:48.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7fa4f3fff700 1 --2- 172.21.15.116:0/4011141353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 0x7fa4f40d11e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa4f406b340 tx=0x7fa4e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2430ed2f7fb99bf server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f63c7f46700 1 Processor -- start 2023-11-08T08:18:48.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f63c7f46700 1 -- start start 2023-11-08T08:18:48.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.613+0000 7f19f3f7e700 1 Processor -- start 2023-11-08T08:18:48.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f19f3f7e700 1 -- start start 2023-11-08T08:18:48.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f19f3f7e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f19ec0cac10 0x7f19ec0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f19f3f7e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0cb160 con 0x7f19ec0ae020 2023-11-08T08:18:48.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f19f3f7e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec06b910 con 0x7f19ec08ac20 2023-11-08T08:18:48.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.614+0000 7f19f3f7e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec06bbf0 con 0x7f19ec0cac10 2023-11-08T08:18:48.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f1d1a700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f19ec0ae020 0x7f19ec06c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34296/0 (socket says 172.21.15.116:34296) 2023-11-08T08:18:48.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f1d1a700 1 -- 172.21.15.116:0/1675425991 learned_addr learned my addr 172.21.15.116:0/1675425991 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f251b700 1 --2- 172.21.15.116:0/1675425991 >> v2:172.21.15.129:3300/0 conn(0x7f19ec0cac10 0x7f19ec0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f63c7f46700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63c006cdd0 0x7f63c014f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f63c7f46700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63c00b0a80 0x7f63c00c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c7f46700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c01632d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c7f46700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c0162040 con 0x7f63c01609c0 2023-11-08T08:18:48.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c7f46700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c0161d60 con 0x7f63c00b0a80 2023-11-08T08:18:48.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c7f46700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c01627f0 con 0x7f63c006cdd0 2023-11-08T08:18:48.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c01632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c01632d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39148/0 (socket says 172.21.15.116:39148) 2023-11-08T08:18:48.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 -- 172.21.15.116:0/997826291 learned_addr learned my addr 172.21.15.116:0/997826291 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c54e1700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63c00b0a80 0x7f63c00c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c5ce2700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63c006cdd0 0x7f63c014f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7fa4f27fc700 1 -- 172.21.15.116:0/4011141353 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4011830 con 0x7fa4f40b5100 2023-11-08T08:18:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7fa4f27fc700 1 -- 172.21.15.116:0/4011141353 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e4011990 con 0x7fa4f40b5100 2023-11-08T08:18:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7fa4f27fc700 1 -- 172.21.15.116:0/4011141353 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4011d20 con 0x7fa4f40b5100 2023-11-08T08:18:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f0d18700 1 -- 172.21.15.116:0/1675425991 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819405065 0 0) 0x7f19ec0cb160 con 0x7f19ec0ae020 2023-11-08T08:18:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f0d18700 1 -- 172.21.15.116:0/1675425991 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d80035b0 con 0x7f19ec0ae020 2023-11-08T08:18:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/4011141353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 msgr2=0x7fa4f40d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7fa4f9ee8700 1 --2- 172.21.15.116:0/4011141353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 0x7fa4f40d11e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa4f406b340 tx=0x7fa4e4009b20 comp rx=0 tx=0).stop 2023-11-08T08:18:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f251b700 1 -- 172.21.15.116:0/1675425991 >> v1:172.21.15.129:6789/0 conn(0x7f19ec08ac20 legacy=0x7f19ec0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f251b700 1 -- 172.21.15.116:0/1675425991 >> v1:172.21.15.116:6789/0 conn(0x7f19ec0ae020 legacy=0x7f19ec06c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.615+0000 7f19f251b700 1 -- 172.21.15.116:0/1675425991 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec06bed0 con 0x7f19ec0cac10 2023-11-08T08:18:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 -- 172.21.15.116:0/997826291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63c006cdd0 msgr2=0x7f63c014f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63c006cdd0 0x7f63c014f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 -- 172.21.15.116:0/997826291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63c00b0a80 msgr2=0x7f63c00c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63c00b0a80 0x7f63c00c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f63c64e3700 1 -- 172.21.15.116:0/997826291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b0009550 con 0x7f63c01609c0 2023-11-08T08:18:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.616+0000 7f19f251b700 1 --2- 172.21.15.116:0/1675425991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 0x7f19ec0b0540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009c80 tx=0x7f19e4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ec3c8d4347abb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7f19f0d18700 1 -- 172.21.15.116:0/1675425991 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400baf0 con 0x7f19ec0cac10 2023-11-08T08:18:48.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/4011141353 shutdown_connections 2023-11-08T08:18:48.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7fa4f9ee8700 1 --2- 172.21.15.116:0/4011141353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 0x7fa4f40d11e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7fa4f9ee8700 1 --2- 172.21.15.116:0/4011141353 >> v2:172.21.15.129:3300/0 conn(0x7fa4f40b8220 0x7fa4f40b6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/4011141353 >> 172.21.15.116:0/4011141353 conn(0x7fa4f40d3c00 msgr2=0x7fa4f406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7f19f0d18700 1 -- 172.21.15.116:0/1675425991 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e400bc50 con 0x7f19ec0cac10 2023-11-08T08:18:48.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7f19f0d18700 1 -- 172.21.15.116:0/1675425991 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e4010480 con 0x7f19ec0cac10 2023-11-08T08:18:48.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7f19f3f7e700 1 -- 172.21.15.116:0/1675425991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 msgr2=0x7f19ec0b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.617+0000 7f19f3f7e700 1 --2- 172.21.15.116:0/1675425991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 0x7f19ec0b0540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009c80 tx=0x7f19e4009f90 comp rx=0 tx=0).stop 2023-11-08T08:18:48.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f2e7a7b9700 1 Processor -- start 2023-11-08T08:18:48.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f68ff21d700 1 Processor -- start 2023-11-08T08:18:48.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f63c64e3700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c01632d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63b800c7a0 tx=0x7f63b800cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8003e10 con 0x7f63c01609c0 2023-11-08T08:18:48.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8005070 con 0x7f63c01609c0 2023-11-08T08:18:48.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8005420 con 0x7f63c01609c0 2023-11-08T08:18:48.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f19f3f7e700 1 -- 172.21.15.116:0/1675425991 shutdown_connections 2023-11-08T08:18:48.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f19f3f7e700 1 --2- 172.21.15.116:0/1675425991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 0x7f19ec0b0540 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f19f3f7e700 1 -- 172.21.15.116:0/1675425991 >> 172.21.15.116:0/1675425991 conn(0x7f19ec06aed0 msgr2=0x7f19ec0cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f2e7a7b9700 1 -- start start 2023-11-08T08:18:48.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e7a7b9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e740ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e7a7b9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2e740adda0 0x7f2e740b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e7a7b9700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e740c2460 con 0x7f2e740b0d20 2023-11-08T08:18:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e7a7b9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e740ae210 con 0x7f2e740adda0 2023-11-08T08:18:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e7a7b9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74161000 con 0x7f2e74160c30 2023-11-08T08:18:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e78d56700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2e740b0d20 0x7f2e740cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34298/0 (socket says 172.21.15.116:34298) 2023-11-08T08:18:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e78d56700 1 -- 172.21.15.116:0/1147338894 learned_addr learned my addr 172.21.15.116:0/1147338894 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c00caff0 con 0x7f63c01609c0 2023-11-08T08:18:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c00cad10 con 0x7f63c01609c0 2023-11-08T08:18:48.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f2e73fff700 1 --2- 172.21.15.116:0/1147338894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e740ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e73fff700 1 -- 172.21.15.116:0/1147338894 >> v2:172.21.15.129:3300/0 conn(0x7f2e740adda0 msgr2=0x7f2e740b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e73fff700 1 --2- 172.21.15.116:0/1147338894 >> v2:172.21.15.129:3300/0 conn(0x7f2e740adda0 0x7f2e740b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e73fff700 1 -- 172.21.15.116:0/1147338894 >> v1:172.21.15.116:6789/0 conn(0x7f2e740b0d20 legacy=0x7f2e740cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e73fff700 1 -- 172.21.15.116:0/1147338894 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7406b770 con 0x7f2e74160c30 2023-11-08T08:18:48.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e73fff700 1 --2- 172.21.15.116:0/1147338894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e740ad860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009d90 tx=0x7f2e6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ab13adf485d7b44 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e72ffd700 1 -- 172.21.15.116:0/1147338894 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c00e070 con 0x7f2e74160c30 2023-11-08T08:18:48.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e72ffd700 1 -- 172.21.15.116:0/1147338894 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c00bbd0 con 0x7f2e74160c30 2023-11-08T08:18:48.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f2e72ffd700 1 -- 172.21.15.116:0/1147338894 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c012430 con 0x7f2e74160c30 2023-11-08T08:18:48.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/1147338894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 msgr2=0x7f2e740ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f2e7a7b9700 1 --2- 172.21.15.116:0/1147338894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e740ad860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009d90 tx=0x7f2e6c00b330 comp rx=0 tx=0).stop 2023-11-08T08:18:48.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.618+0000 7f68ff21d700 1 -- start start 2023-11-08T08:18:48.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68ff21d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f68f8158980 0x7f68f80adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68ff21d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68f806d230 0x7f68f806b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68ff21d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f80b0540 0x7f68f80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68ff21d700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f806c700 con 0x7f68f806d230 2023-11-08T08:18:48.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68ff21d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f80b0ec0 con 0x7f68f8158980 2023-11-08T08:18:48.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68ff21d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f806d6c0 con 0x7f68f80b0540 2023-11-08T08:18:48.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68fcfb9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f68f8158980 0x7f68f80adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68fcfb9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f68f8158980 0x7f68f80adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57048/0 (socket says 172.21.15.116:57048) 2023-11-08T08:18:48.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f68fcfb9700 1 -- 172.21.15.116:0/1760351669 learned_addr learned my addr 172.21.15.116:0/1760351669 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68fcfb9700 1 -- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f80b0540 msgr2=0x7f68f80b0980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:18:48.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68fcfb9700 1 --2- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f80b0540 0x7f68f80b0980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68fcfb9700 1 -- 172.21.15.116:0/1760351669 >> v2:172.21.15.116:3300/0 conn(0x7f68f806d230 msgr2=0x7f68f806b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68fcfb9700 1 --2- 172.21.15.116:0/1760351669 >> v2:172.21.15.116:3300/0 conn(0x7f68f806d230 0x7f68f806b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68fcfb9700 1 -- 172.21.15.116:0/1760351669 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f80ca7f0 con 0x7f68f8158980 2023-11-08T08:18:48.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68fcfb9700 1 --2- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f80adf60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f68f8071f90 tx=0x7f68e8009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f6cba61d8cfc0de server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68f77fe700 1 -- 172.21.15.116:0/1760351669 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e8013c90 con 0x7f68f8158980 2023-11-08T08:18:48.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68f77fe700 1 -- 172.21.15.116:0/1760351669 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e8013df0 con 0x7f68f8158980 2023-11-08T08:18:48.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.620+0000 7f68f77fe700 1 -- 172.21.15.116:0/1760351669 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e800b680 con 0x7f68f8158980 2023-11-08T08:18:48.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 -- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 msgr2=0x7f68f80adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 --2- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f80adf60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f68f8071f90 tx=0x7f68e8009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63c0162670 con 0x7f63c01609c0 2023-11-08T08:18:48.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/4011141353 shutdown_connections 2023-11-08T08:18:48.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.619+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/4011141353 wait complete. 2023-11-08T08:18:48.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7fa4f9ee8700 1 Processor -- start 2023-11-08T08:18:48.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f19f3f7e700 1 -- 172.21.15.116:0/1675425991 shutdown_connections 2023-11-08T08:18:48.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f245831a700 1 Processor -- start 2023-11-08T08:18:48.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f63b8010070 con 0x7f63c01609c0 2023-11-08T08:18:48.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f63b67fc700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63a40682f0 0x7f63a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f63b8085000 con 0x7f63c01609c0 2023-11-08T08:18:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f63b67fc700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f63a4070b20 0x7f63a4072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 --> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63a40735e0 con 0x7f63a4070b20 2023-11-08T08:18:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63b8085460 con 0x7f63c01609c0 2023-11-08T08:18:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f63c54e1700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f63a4070b20 0x7f63a4072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 -- start start 2023-11-08T08:18:48.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4f40b25d0 0x7fa4f40cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 0x7fa4f40ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4f40b8220 0x7fa4f40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f40cb260 con 0x7fa4f40b5100 2023-11-08T08:18:48.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4169d60 con 0x7fa4f40b8220 2023-11-08T08:18:48.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7fa4f9ee8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4169150 con 0x7fa4f40b25d0 2023-11-08T08:18:48.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/1147338894 shutdown_connections 2023-11-08T08:18:48.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f2e7a7b9700 1 --2- 172.21.15.116:0/1147338894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e740ad860 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f2e7a7b9700 1 --2- 172.21.15.116:0/1147338894 >> v2:172.21.15.129:3300/0 conn(0x7f2e740adda0 0x7f2e740b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/1147338894 >> 172.21.15.116:0/1147338894 conn(0x7f2e740cb540 msgr2=0x7f2e740cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 -- 172.21.15.116:0/1760351669 shutdown_connections 2023-11-08T08:18:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 --2- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f80b0540 0x7f68f80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 --2- 172.21.15.116:0/1760351669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f80adf60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 --2- 172.21.15.116:0/1760351669 >> v2:172.21.15.116:3300/0 conn(0x7f68f806d230 0x7f68f806b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f68ff21d700 1 -- 172.21.15.116:0/1760351669 >> 172.21.15.116:0/1760351669 conn(0x7f68f806aed0 msgr2=0x7f68f80cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f63c54e1700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f63a4070b20 0x7f63a4072f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f63c5ce2700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63a40682f0 0x7f63a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4f40b8220 0x7fa4f40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4f40b8220 0x7fa4f40cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57064/0 (socket says 172.21.15.116:57064) 2023-11-08T08:18:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 -- 172.21.15.116:0/2917450879 learned_addr learned my addr 172.21.15.116:0/2917450879 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.621+0000 7f245831a700 1 -- start start 2023-11-08T08:18:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.622+0000 7f245831a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f245006a980 0x7f24500cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f245831a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f245015c510 0x7f2450070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f245831a700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24500700f0 con 0x7f245015c020 2023-11-08T08:18:48.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f245831a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245015cc60 con 0x7f245006a980 2023-11-08T08:18:48.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f245831a700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245015cf40 con 0x7f245015c510 2023-11-08T08:18:48.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f24560b6700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f245015c020 0x7f245006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34306/0 (socket says 172.21.15.116:34306) 2023-11-08T08:18:48.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f24560b6700 1 -- 172.21.15.116:0/562496301 learned_addr learned my addr 172.21.15.116:0/562496301 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f19f3f7e700 1 -- 172.21.15.116:0/1675425991 wait complete. 2023-11-08T08:18:48.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 -- 172.21.15.116:0/2917450879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4f40b25d0 msgr2=0x7fa4f40cc650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-08T08:18:48.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4f40b25d0 0x7fa4f40cc650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 -- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 msgr2=0x7fa4f40ccb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-08T08:18:48.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f40b5100 0x7fa4f40ccb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123719089 0 0) 0x7f24500700f0 con 0x7f245015c020 2023-11-08T08:18:48.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24400035b0 con 0x7f245015c020 2023-11-08T08:18:48.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 -- 172.21.15.116:0/2917450879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e40094a0 con 0x7fa4f40b8220 2023-11-08T08:18:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7fa4f3fff700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4f40b8220 0x7fa4f40cf980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4f406be70 tx=0x7fa4e4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4004590 con 0x7fa4f40b8220 2023-11-08T08:18:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e401f070 con 0x7fa4f40b8220 2023-11-08T08:18:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40038c0 con 0x7fa4f40b8220 2023-11-08T08:18:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359457769 0 0) 0x7f24400035b0 con 0x7f245015c020 2023-11-08T08:18:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 >> v2:172.21.15.129:3300/0 conn(0x7f245015c510 msgr2=0x7f2450070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24568b7700 1 --2- 172.21.15.116:0/562496301 >> v2:172.21.15.129:3300/0 conn(0x7f245015c510 0x7f2450070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 --2- 172.21.15.116:0/562496301 >> v2:172.21.15.129:3300/0 conn(0x7f245015c510 0x7f2450070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 >> v2:172.21.15.116:3300/0 conn(0x7f245006a980 msgr2=0x7f24500cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 --2- 172.21.15.116:0/562496301 >> v2:172.21.15.116:3300/0 conn(0x7f245006a980 0x7f24500cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245015d220 con 0x7f245015c020 2023-11-08T08:18:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f24440019f0 con 0x7f245015c020 2023-11-08T08:18:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 >> v1:172.21.15.116:6789/0 conn(0x7f245015c020 legacy=0x7f245006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2440004be0 0x7f24400037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440003ff0 0x7f2440009010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2440009550 0x7f24400099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245015cc60 con 0x7f2440009550 2023-11-08T08:18:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245015cf40 con 0x7f2440003ff0 2023-11-08T08:18:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244000a270 con 0x7f2440004be0 2023-11-08T08:18:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f19f3f7e700 1 Processor -- start 2023-11-08T08:18:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f68ff21d700 1 -- 172.21.15.116:0/1760351669 shutdown_connections 2023-11-08T08:18:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24560b6700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2440004be0 0x7f24400037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f24568b7700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2440009550 0x7f24400099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24560b6700 1 -- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440003ff0 msgr2=0x7f2440009010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-08T08:18:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24560b6700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440003ff0 0x7f2440009010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24560b6700 1 -- 172.21.15.116:0/562496301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2440009550 msgr2=0x7f24400099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24560b6700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2440009550 0x7f24400099f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24560b6700 1 -- 172.21.15.116:0/562496301 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245015d220 con 0x7f2440004be0 2023-11-08T08:18:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24568b7700 1 -- 172.21.15.116:0/562496301 reap_dead start 2023-11-08T08:18:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f24560b6700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2440004be0 0x7f24400037f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f244400a8c0 tx=0x7f2440004870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a82e9bd3f73940fc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7ff70be8e700 1 Processor -- start 2023-11-08T08:18:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444012600 con 0x7f2440004be0 2023-11-08T08:18:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2444012780 con 0x7f2440004be0 2023-11-08T08:18:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f24550b4700 1 -- 172.21.15.116:0/562496301 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444012b30 con 0x7f2440004be0 2023-11-08T08:18:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f63c5ce2700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63a40682f0 0x7f63a406a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f63b0000c00 tx=0x7f63b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== osd.2 v2:172.21.15.116:6816/121156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f63a40735e0 con 0x7f63a4070b20 2023-11-08T08:18:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.623+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/1147338894 shutdown_connections 2023-11-08T08:18:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/1147338894 wait complete. 2023-11-08T08:18:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.624+0000 7f19f3f7e700 1 -- start start 2023-11-08T08:18:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f19f3f7e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ec0ae020 0x7f19ec167060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f3f7e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 0x7f19ec0c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f3f7e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19ec0c8730 0x7f19ec0b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f3f7e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0c7dd0 con 0x7f19ec0ae020 2023-11-08T08:18:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f2e7a7b9700 1 Processor -- start 2023-11-08T08:18:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f3f7e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec1675a0 con 0x7f19ec0cac10 2023-11-08T08:18:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f3f7e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0c7150 con 0x7f19ec0c8730 2023-11-08T08:18:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ec0ae020 0x7f19ec167060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ec0ae020 0x7f19ec167060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39188/0 (socket says 172.21.15.116:39188) 2023-11-08T08:18:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 -- 172.21.15.116:0/4261134187 learned_addr learned my addr 172.21.15.116:0/4261134187 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1519700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 0x7f19ec0c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f251b700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19ec0c8730 0x7f19ec0b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7ff70be8e700 1 -- start start 2023-11-08T08:18:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 -- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19ec0c8730 msgr2=0x7f19ec0b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19ec0c8730 0x7f19ec0b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 -- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 msgr2=0x7f19ec0c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ec0cac10 0x7f19ec0c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f19f1d1a700 1 -- 172.21.15.116:0/4261134187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e400d040 con 0x7f19ec0ae020 2023-11-08T08:18:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f4071d50 con 0x7fa4f40b8220 2023-11-08T08:18:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f40d0ce0 con 0x7fa4f40b8220 2023-11-08T08:18:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.625+0000 7f68ff21d700 1 -- 172.21.15.116:0/1760351669 wait complete. 2023-11-08T08:18:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f68ff21d700 1 Processor -- start 2023-11-08T08:18:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f19f1d1a700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ec0ae020 0x7f19ec167060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19e800bc50 tx=0x7f19e800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 -- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2440004be0 msgr2=0x7f24400037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2440004be0 0x7f24400037f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f244400a8c0 tx=0x7f2440004870 comp rx=0 tx=0).stop 2023-11-08T08:18:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 -- 172.21.15.116:0/562496301 shutdown_connections 2023-11-08T08:18:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 --2- 172.21.15.116:0/562496301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2440004be0 0x7f24400037f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 -- 172.21.15.116:0/562496301 >> 172.21.15.116:0/562496301 conn(0x7f24500b16c0 msgr2=0x7f24500cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 -- 172.21.15.116:0/562496301 shutdown_connections 2023-11-08T08:18:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e800cbc0 con 0x7f19ec0ae020 2023-11-08T08:18:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e800cd20 con 0x7f19ec0ae020 2023-11-08T08:18:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec0ccda0 con 0x7f19ec0ae020 2023-11-08T08:18:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e80156b0 con 0x7f19ec0ae020 2023-11-08T08:18:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec06cb00 con 0x7f19ec0ae020 2023-11-08T08:18:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.626+0000 7f2e7a7b9700 1 -- start start 2023-11-08T08:18:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f2e7a7b9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e740adda0 0x7f2e74163490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f2e7a7b9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e740b0d20 0x7f2e741639d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f2e7a7b9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e74202770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f2e7a7b9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74160870 con 0x7f2e740adda0 2023-11-08T08:18:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f2e7a7b9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74163f10 con 0x7f2e740b0d20 2023-11-08T08:18:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f2e7a7b9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74163000 con 0x7f2e74160c30 2023-11-08T08:18:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f2e73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e740adda0 0x7f2e74163490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f2e73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e740adda0 0x7f2e74163490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39196/0 (socket says 172.21.15.116:39196) 2023-11-08T08:18:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f2e73fff700 1 -- 172.21.15.116:0/2912212319 learned_addr learned my addr 172.21.15.116:0/2912212319 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f68ff21d700 1 -- start start 2023-11-08T08:18:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68ff21d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f806d230 0x7f68f81f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68ff21d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f80b0540 0x7f68f81f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68ff21d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f81ffb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68ff21d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80b2030 con 0x7f68f80b0540 2023-11-08T08:18:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68ff21d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80b2310 con 0x7f68f8158980 2023-11-08T08:18:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68ff21d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80b25f0 con 0x7f68f806d230 2023-11-08T08:18:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.628+0000 7f68fcfb9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f806d230 0x7f68f81f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fd7ba700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f81ffb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fd7ba700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f81ffb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57088/0 (socket says 172.21.15.116:57088) 2023-11-08T08:18:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fd7ba700 1 -- 172.21.15.116:0/3216238061 learned_addr learned my addr 172.21.15.116:0/3216238061 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68f7fff700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f80b0540 0x7f68f81f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f2e73fff700 1 -- 172.21.15.116:0/2912212319 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 msgr2=0x7f2e74202770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f2e73fff700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e74160c30 0x7f2e74202770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f2e73fff700 1 -- 172.21.15.116:0/2912212319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e740b0d20 msgr2=0x7f2e741639d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-08T08:18:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f2e73fff700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e740b0d20 0x7f2e741639d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f2e73fff700 1 -- 172.21.15.116:0/2912212319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c009a70 con 0x7f2e740adda0 2023-11-08T08:18:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7f245831a700 1 -- 172.21.15.116:0/562496301 wait complete. 2023-11-08T08:18:48.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f245831a700 1 Processor -- start 2023-11-08T08:18:48.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fcfb9700 1 -- 172.21.15.116:0/3216238061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 msgr2=0x7f68f81ffb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fcfb9700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f8158980 0x7f68f81ffb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fcfb9700 1 -- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f80b0540 msgr2=0x7f68f81f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fcfb9700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f80b0540 0x7f68f81f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7f68fcfb9700 1 -- 172.21.15.116:0/3216238061 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e8009550 con 0x7f68f806d230 2023-11-08T08:18:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7ff70be8e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff704160e60 0x7ff7040cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7ff70be8e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff70406c820 0x7ff7040717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7ff70be8e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040ced90 con 0x7ff704161230 2023-11-08T08:18:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7ff70be8e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7040cff00 con 0x7ff70406c820 2023-11-08T08:18:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.629+0000 7ff70be8e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70406c390 con 0x7ff704160e60 2023-11-08T08:18:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff709429700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff704161230 0x7ff7040ca970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57274/0 (socket says 172.21.15.116:57274) 2023-11-08T08:18:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff709429700 1 -- 172.21.15.116:0/2367797846 learned_addr learned my addr 172.21.15.116:0/2367797846 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f68fcfb9700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f806d230 0x7f68f81f2010 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f68e801a000 tx=0x7f68e8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff70a42b700 1 --2- 172.21.15.116:0/2367797846 >> v2:172.21.15.116:3300/0 conn(0x7ff70406c820 0x7ff7040717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.627+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa4f4071ee0 con 0x7fa4f40b8220 2023-11-08T08:18:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff708c28700 1 -- 172.21.15.116:0/2367797846 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268013301 0 0) 0x7ff7040ced90 con 0x7ff704161230 2023-11-08T08:18:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff708c28700 1 -- 172.21.15.116:0/2367797846 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f00035b0 con 0x7ff704161230 2023-11-08T08:18:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f2e73fff700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e740adda0 0x7f2e74163490 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c014040 tx=0x7f2e6c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e8013640 con 0x7f68f806d230 2023-11-08T08:18:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e8004030 con 0x7f68f806d230 2023-11-08T08:18:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e80043c0 con 0x7f68f806d230 2023-11-08T08:18:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f80cadb0 con 0x7f68f806d230 2023-11-08T08:18:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f80caad0 con 0x7f68f806d230 2023-11-08T08:18:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff70a42b700 1 -- 172.21.15.116:0/2367797846 >> v2:172.21.15.129:3300/0 conn(0x7ff704160e60 msgr2=0x7ff7040cf9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:18:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff70a42b700 1 --2- 172.21.15.116:0/2367797846 >> v2:172.21.15.129:3300/0 conn(0x7ff704160e60 0x7ff7040cf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff70a42b700 1 -- 172.21.15.116:0/2367797846 >> v1:172.21.15.129:6789/0 conn(0x7ff704161230 legacy=0x7ff7040ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff70a42b700 1 -- 172.21.15.116:0/2367797846 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7040cb150 con 0x7ff70406c820 2023-11-08T08:18:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7ff70a42b700 1 --2- 172.21.15.116:0/2367797846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040717f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff70406d420 tx=0x7ff6fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a69b45279d61d28 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f19e8015810 con 0x7f19ec0ae020 2023-11-08T08:18:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.630+0000 7f19e27fc700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19d00682f0 0x7f19d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7ff708c28700 1 -- 172.21.15.116:0/2367797846 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc00e070 con 0x7ff70406c820 2023-11-08T08:18:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa4e400f050 con 0x7fa4f40b8220 2023-11-08T08:18:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f19e8086060 con 0x7f19ec0ae020 2023-11-08T08:18:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f19f1519700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19d00682f0 0x7f19d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c009410 con 0x7f2e740adda0 2023-11-08T08:18:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c020070 con 0x7f2e740adda0 2023-11-08T08:18:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c004280 con 0x7f2e740adda0 2023-11-08T08:18:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e74164a90 con 0x7f2e740adda0 2023-11-08T08:18:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e741647b0 con 0x7f2e740adda0 2023-11-08T08:18:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e74164d70 con 0x7f2e740adda0 2023-11-08T08:18:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.632+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68e8021070 con 0x7f68f806d230 2023-11-08T08:18:48.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68f80cb090 con 0x7f68f806d230 2023-11-08T08:18:48.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7fa4dbfff700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4dc068450 0x7fa4dc06a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.631+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa4e4021070 con 0x7fa4f40b8220 2023-11-08T08:18:48.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.632+0000 7fa4dbfff700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7fa4dc070c00 0x7fa4dc073010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.632+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4dc0736c0 con 0x7fa4dc070c00 2023-11-08T08:18:48.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.632+0000 7fa4f37fe700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4dc068450 0x7fa4dc06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7fa4f2ffd700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7fa4dc070c00 0x7fa4dc073010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7fa4f37fe700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4dc068450 0x7fa4dc06a900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80099d0 tx=0x7fa4e8009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f19f1519700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19d00682f0 0x7f19d006a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0cd9d0 tx=0x7f19dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.632+0000 7ff708c28700 1 -- 172.21.15.116:0/2367797846 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc00bae0 con 0x7ff70406c820 2023-11-08T08:18:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.632+0000 7ff708c28700 1 -- 172.21.15.116:0/2367797846 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc00be50 con 0x7ff70406c820 2023-11-08T08:18:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 -- 172.21.15.116:0/2367797846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 msgr2=0x7ff7040717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 --2- 172.21.15.116:0/2367797846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040717f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff70406d420 tx=0x7ff6fc00b330 comp rx=0 tx=0).stop 2023-11-08T08:18:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4e4053ce0 con 0x7fa4f40b8220 2023-11-08T08:18:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f245831a700 1 -- start start 2023-11-08T08:18:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f19f3f7e700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f19ec08ac20 0x7f19ec0c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 --> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19ec0c7bb0 con 0x7f19ec08ac20 2023-11-08T08:18:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 -- 172.21.15.116:0/2367797846 shutdown_connections 2023-11-08T08:18:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 --2- 172.21.15.116:0/2367797846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040717f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 --2- 172.21.15.116:0/2367797846 >> v2:172.21.15.129:3300/0 conn(0x7ff704160e60 0x7ff7040cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 -- 172.21.15.116:0/2367797846 >> 172.21.15.116:0/2367797846 conn(0x7ff7040b16c0 msgr2=0x7ff7040cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7f19f251b700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f19ec08ac20 0x7f19ec0c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7ff70be8e700 1 -- 172.21.15.116:0/2367797846 shutdown_connections 2023-11-08T08:18:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e6c0043e0 con 0x7f2e740adda0 2023-11-08T08:18:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f245831a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440004be0 0x7f24501fea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f245831a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24500b1d50 0x7f24500c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f245831a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24500c6530 0x7f2450154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f245831a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24500c7650 con 0x7f24500c6530 2023-11-08T08:18:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f245831a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24500c6a40 con 0x7f2440004be0 2023-11-08T08:18:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f245831a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24500c6d20 con 0x7f24500b1d50 2023-11-08T08:18:48.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.633+0000 7fa4f2ffd700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7fa4dc070c00 0x7fa4dc073010 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f2e70ff9700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e60068340 0x7f2e6006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== osd.3 v2:172.21.15.116:6808/121155 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa4dc0736c0 con 0x7fa4dc070c00 2023-11-08T08:18:48.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2e6c022070 con 0x7f2e740adda0 2023-11-08T08:18:48.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f2e70ff9700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7f2e60070bf0 0x7f2e60073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 --> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e600736b0 con 0x7f2e60070bf0 2023-11-08T08:18:48.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e6c08a820 con 0x7f2e740adda0 2023-11-08T08:18:48.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7f2e737fe700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e60068340 0x7f2e6006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7f2e737fe700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e60068340 0x7f2e6006a7f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e7406feb0 tx=0x7f2e64006cb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7f24568b7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24500c6530 0x7f2450154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7f24568b7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24500c6530 0x7f2450154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39216/0 (socket says 172.21.15.116:39216) 2023-11-08T08:18:48.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7f24568b7700 1 -- 172.21.15.116:0/2128227291 learned_addr learned my addr 172.21.15.116:0/2128227291 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7f24560b6700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440004be0 0x7f24501fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f24568b7700 1 -- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24500b1d50 msgr2=0x7f24500c5ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:48.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f24568b7700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24500b1d50 0x7f24500c5ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f24568b7700 1 -- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440004be0 msgr2=0x7f24501fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f24568b7700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2440004be0 0x7f24501fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f24568b7700 1 -- 172.21.15.116:0/2128227291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2450068450 con 0x7f24500c6530 2023-11-08T08:18:48.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f24568b7700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24500c6530 0x7f2450154fa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f24500c5f80 tx=0x7f244800fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448013910 con 0x7f24500c6530 2023-11-08T08:18:48.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244800c040 con 0x7f24500c6530 2023-11-08T08:18:48.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448013cb0 con 0x7f24500c6530 2023-11-08T08:18:48.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f245015c930 con 0x7f24500c6530 2023-11-08T08:18:48.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.636+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24500685e0 con 0x7f24500c6530 2023-11-08T08:18:48.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.634+0000 7ff70be8e700 1 -- 172.21.15.116:0/2367797846 wait complete. 2023-11-08T08:18:48.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.635+0000 7ff70be8e700 1 Processor -- start 2023-11-08T08:18:48.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.637+0000 7f19f251b700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f19ec08ac20 0x7f19ec0c7670 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.637+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f24500743f0 con 0x7f24500c6530 2023-11-08T08:18:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.637+0000 7f2e78d56700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7f2e60070bf0 0x7f2e60073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7f2e78d56700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7f2e60070bf0 0x7f2e60073000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7f68f57fa700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68e40683c0 0x7f68e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7f68f7fff700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68e40683c0 0x7f68e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f68e8087ab0 con 0x7f68f806d230 2023-11-08T08:18:48.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7f68f57fa700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f68e4070bf0 0x7f68e4073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 --> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e40736b0 con 0x7f68e4070bf0 2023-11-08T08:18:48.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f68fd7ba700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f68e4070bf0 0x7f68e4073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f68fd7ba700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f68e4070bf0 0x7f68e4073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68e80542a0 con 0x7f68f806d230 2023-11-08T08:18:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f68f7fff700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68e40683c0 0x7f68e406a870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68f80c5610 tx=0x7f68ec009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== osd.4 v2:172.21.15.129:6800/121420 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f19ec0c7bb0 con 0x7f19ec08ac20 2023-11-08T08:18:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2448005a20 con 0x7f24500c6530 2023-11-08T08:18:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f244effd700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2434068340 0x7f243406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2448086640 con 0x7f24500c6530 2023-11-08T08:18:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f244effd700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f2434070bf0 0x7f2434073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 --> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24340736b0 con 0x7f2434070bf0 2023-11-08T08:18:48.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2448088d10 con 0x7f24500c6530 2023-11-08T08:18:48.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7f24558b5700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f2434070bf0 0x7f2434073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.638+0000 7ff70be8e700 1 -- start start 2023-11-08T08:18:48.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7ff70be8e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7ff70be8e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff704160e60 0x7ff70415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7ff70be8e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff704161230 0x7ff70406a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7ff70be8e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff704161600 con 0x7ff70406c820 2023-11-08T08:18:48.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7ff70be8e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040b32e0 con 0x7ff704161230 2023-11-08T08:18:48.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.639+0000 7ff70be8e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040b4c50 con 0x7ff704160e60 2023-11-08T08:18:48.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff704161230 0x7ff70406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff704161230 0x7ff70406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57118/0 (socket says 172.21.15.116:57118) 2023-11-08T08:18:48.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 -- 172.21.15.116:0/16915994 learned_addr learned my addr 172.21.15.116:0/16915994 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff709429700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff704160e60 0x7ff70415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff709c2a700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 -- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff704160e60 msgr2=0x7ff70415e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff704160e60 0x7ff70415e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 -- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 msgr2=0x7ff7040b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff70a42b700 1 -- 172.21.15.116:0/16915994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc009a70 con 0x7ff704161230 2023-11-08T08:18:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.640+0000 7ff709c2a700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70406c820 0x7ff7040b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.641+0000 7ff70a42b700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff704161230 0x7ff70406a270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00b370 tx=0x7ff6fc004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.641+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== osd.1 v2:172.21.15.116:6800/121158 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f68e40736b0 con 0x7f68e4070bf0 2023-11-08T08:18:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.641+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc004950 con 0x7ff704161230 2023-11-08T08:18:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.641+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc009410 con 0x7ff704161230 2023-11-08T08:18:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.641+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc012650 con 0x7ff704161230 2023-11-08T08:18:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.642+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7040cb710 con 0x7ff704161230 2023-11-08T08:18:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.642+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7040cb430 con 0x7ff704161230 2023-11-08T08:18:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.644+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6fc02d070 con 0x7ff704161230 2023-11-08T08:18:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.644+0000 7ff6fa7fc700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6e80682c0 0x7ff6e806a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.645+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff6fc087890 con 0x7ff704161230 2023-11-08T08:18:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.646+0000 7ff70be8e700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7ff70415ffa0 0x7ff7040c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.646+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7040cf5c0 con 0x7ff70415ffa0 2023-11-08T08:18:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.647+0000 7f24558b5700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f2434070bf0 0x7f2434073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.647+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== osd.5 v2:172.21.15.129:6808/121421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f24340736b0 con 0x7f2434070bf0 2023-11-08T08:18:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.648+0000 7ff709429700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7ff70415ffa0 0x7ff7040c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.648+0000 7ff709429700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7ff70415ffa0 0x7ff7040c3a20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.648+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 --> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f63c00cb2d0 con 0x7f63a4070b20 2023-11-08T08:18:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.648+0000 7f24560b6700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2434068340 0x7f243406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.641+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== osd.0 v2:172.21.15.116:6824/121157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2e600736b0 con 0x7f2e60070bf0 2023-11-08T08:18:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.649+0000 7ff709c2a700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6e80682c0 0x7ff6e806a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.649+0000 7f63b67fc700 1 -- 172.21.15.116:0/997826291 <== osd.2 v2:172.21.15.116:6816/121156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f63c00cb2d0 con 0x7f63a4070b20 2023-11-08T08:18:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.649+0000 7ff709c2a700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6e80682c0 0x7ff6e806a770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006fd0 tx=0x7ff6f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.649+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f63a4070b20 msgr2=0x7f63a4072f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.649+0000 7f63c7f46700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f63a4070b20 0x7f63a4072f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.650+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63a40682f0 msgr2=0x7f63a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.650+0000 7f63c7f46700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63a40682f0 0x7f63a406a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f63b0000c00 tx=0x7f63b0008030 comp rx=0 tx=0).stop 2023-11-08T08:18:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.650+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 msgr2=0x7f63c01632d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.650+0000 7f63c7f46700 1 --2- 172.21.15.116:0/997826291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c01609c0 0x7f63c01632d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63b800c7a0 tx=0x7f63b800cab0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.650+0000 7f63c64e3700 1 -- 172.21.15.116:0/997826291 reap_dead start 2023-11-08T08:18:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.651+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== osd.7 v2:172.21.15.129:6816/121422 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff7040cf5c0 con 0x7ff70415ffa0 2023-11-08T08:18:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.651+0000 7f24560b6700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2434068340 0x7f243406a7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2444000f80 tx=0x7f2444009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.651+0000 7fdc4df1f700 1 Processor -- start 2023-11-08T08:18:48.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.652+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 shutdown_connections 2023-11-08T08:18:48.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.652+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 >> 172.21.15.116:0/997826291 conn(0x7f63c006aed0 msgr2=0x7f63c00ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.658+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 shutdown_connections 2023-11-08T08:18:48.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.658+0000 7f63c7f46700 1 -- 172.21.15.116:0/997826291 wait complete. 2023-11-08T08:18:48.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.658+0000 7fdc4df1f700 1 -- start start 2023-11-08T08:18:48.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.659+0000 7fdc4df1f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc48160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.659+0000 7fdc4df1f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdc4806b220 0x7fdc480d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.659+0000 7fdc4df1f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480d2d30 con 0x7fdc480b5dc0 2023-11-08T08:18:48.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.659+0000 7fdc4df1f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc480d0b40 con 0x7fdc4806b220 2023-11-08T08:18:48.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.659+0000 7fdc4df1f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc481611f0 con 0x7fdc4815c8c0 2023-11-08T08:18:48.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc46ffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fdc480b5dc0 0x7fdc4806ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57288/0 (socket says 172.21.15.116:57288) 2023-11-08T08:18:48.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc46ffd700 1 -- 172.21.15.116:0/316284329 learned_addr learned my addr 172.21.15.116:0/316284329 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:48.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc477fe700 1 --2- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc48160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc47fff700 1 --2- 172.21.15.116:0/316284329 >> v2:172.21.15.129:3300/0 conn(0x7fdc4806b220 0x7fdc480d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4f40743f0 con 0x7fa4dc070c00 2023-11-08T08:18:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc467fc700 1 -- 172.21.15.116:0/316284329 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628941656 0 0) 0x7fdc3c001110 con 0x7fdc480b5dc0 2023-11-08T08:18:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc467fc700 1 -- 172.21.15.116:0/316284329 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc300035b0 con 0x7fdc480b5dc0 2023-11-08T08:18:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc47fff700 1 -- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 msgr2=0x7fdc48160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc47fff700 1 --2- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc48160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc47fff700 1 -- 172.21.15.116:0/316284329 >> v1:172.21.15.129:6789/0 conn(0x7fdc480b5dc0 legacy=0x7fdc4806ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.660+0000 7fdc47fff700 1 -- 172.21.15.116:0/316284329 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480d2a50 con 0x7fdc4806b220 2023-11-08T08:18:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4dbfff700 1 -- 172.21.15.116:0/2917450879 <== osd.3 v2:172.21.15.116:6808/121155 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa4f40743f0 con 0x7fa4dc070c00 2023-11-08T08:18:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7fa4dc070c00 msgr2=0x7fa4dc073010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f9ee8700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7fa4dc070c00 0x7fa4dc073010 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4dc068450 msgr2=0x7fa4dc06a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f9ee8700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4dc068450 0x7fa4dc06a900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80099d0 tx=0x7fa4e8009210 comp rx=0 tx=0).stop 2023-11-08T08:18:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4f40b8220 msgr2=0x7fa4f40cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f9ee8700 1 --2- 172.21.15.116:0/2917450879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4f40b8220 0x7fa4f40cf980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4f406be70 tx=0x7fa4e4004750 comp rx=0 tx=0).stop 2023-11-08T08:18:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.661+0000 7fa4f3fff700 1 -- 172.21.15.116:0/2917450879 reap_dead start 2023-11-08T08:18:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 shutdown_connections 2023-11-08T08:18:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 >> 172.21.15.116:0/2917450879 conn(0x7fa4f40d3c00 msgr2=0x7fa4f40cd410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fdc47fff700 1 --2- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc480d0fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009840 tx=0x7fdc38009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22a199ac345a168a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 shutdown_connections 2023-11-08T08:18:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fdc467fc700 1 -- 172.21.15.116:0/316284329 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38011830 con 0x7fdc4806b220 2023-11-08T08:18:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fdc467fc700 1 -- 172.21.15.116:0/316284329 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc38011990 con 0x7fdc4806b220 2023-11-08T08:18:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fdc467fc700 1 -- 172.21.15.116:0/316284329 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38011d20 con 0x7fdc4806b220 2023-11-08T08:18:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.663+0000 7fa4f9ee8700 1 -- 172.21.15.116:0/2917450879 wait complete. 2023-11-08T08:18:48.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 -- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 msgr2=0x7fdc480d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc480d0fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009840 tx=0x7fdc38009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 -- 172.21.15.116:0/316284329 shutdown_connections 2023-11-08T08:18:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc480d0fd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/316284329 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc48160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 -- 172.21.15.116:0/316284329 >> 172.21.15.116:0/316284329 conn(0x7fdc4806c520 msgr2=0x7fdc4806c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 -- 172.21.15.116:0/316284329 shutdown_connections 2023-11-08T08:18:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.664+0000 7fdc4df1f700 1 -- 172.21.15.116:0/316284329 wait complete. 2023-11-08T08:18:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.665+0000 7fdc4df1f700 1 Processor -- start 2023-11-08T08:18:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.665+0000 7fdc4df1f700 1 -- start start 2023-11-08T08:18:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc4df1f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc481fc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc4df1f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc480b5dc0 0x7fdc48200030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc4df1f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc480c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc4df1f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc481fee70 con 0x7fdc480b5dc0 2023-11-08T08:18:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc4df1f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480b6610 con 0x7fdc4806b220 2023-11-08T08:18:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc4df1f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480b68f0 con 0x7fdc4815c8c0 2023-11-08T08:18:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc477fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc481fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc477fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc481fc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57136/0 (socket says 172.21.15.116:57136) 2023-11-08T08:18:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.666+0000 7fdc477fe700 1 -- 172.21.15.116:0/3834660530 learned_addr learned my addr 172.21.15.116:0/3834660530 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc47fff700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc480c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc477fe700 1 -- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 msgr2=0x7fdc480c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc477fe700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc4815c8c0 0x7fdc480c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc477fe700 1 -- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc480b5dc0 msgr2=0x7fdc48200030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc477fe700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc480b5dc0 0x7fdc48200030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc477fe700 1 -- 172.21.15.116:0/3834660530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc380094a0 con 0x7fdc4806b220 2023-11-08T08:18:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc477fe700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc481fc470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000efc0 tx=0x7fdc4000c0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4000cc00 con 0x7fdc4806b220 2023-11-08T08:18:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4000f040 con 0x7fdc4806b220 2023-11-08T08:18:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40014480 con 0x7fdc4806b220 2023-11-08T08:18:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc480cda40 con 0x7fdc4806b220 2023-11-08T08:18:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.667+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc480cd760 con 0x7fdc4806b220 2023-11-08T08:18:48.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.669+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc400145e0 con 0x7fdc4806b220 2023-11-08T08:18:48.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.669+0000 7fdc2ffff700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdc340682f0 0x7fdc3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.670+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdc400869f0 con 0x7fdc4806b220 2023-11-08T08:18:48.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.670+0000 7fdc46ffd700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdc340682f0 0x7fdc3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.671+0000 7fdc46ffd700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdc340682f0 0x7fdc3406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005910 tx=0x7fdc3c001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.672+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 --> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f19ec0743f0 con 0x7f19ec08ac20 2023-11-08T08:18:48.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.674+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 --> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e74163140 con 0x7f2e60070bf0 2023-11-08T08:18:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.674+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7fdc480c8b10 0x7fdc481603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.674+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 --> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc481ff110 con 0x7fdc480c8b10 2023-11-08T08:18:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.674+0000 7f19e27fc700 1 -- 172.21.15.116:0/4261134187 <== osd.4 v2:172.21.15.129:6800/121420 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f19ec0743f0 con 0x7f19ec08ac20 2023-11-08T08:18:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.674+0000 7fdc47fff700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7fdc480c8b10 0x7fdc481603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.675+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 --> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68f80743f0 con 0x7f68e4070bf0 2023-11-08T08:18:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.675+0000 7fdc47fff700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7fdc480c8b10 0x7fdc481603a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e70ff9700 1 -- 172.21.15.116:0/2912212319 <== osd.0 v2:172.21.15.116:6824/121157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e68001240 con 0x7f2e60070bf0 2023-11-08T08:18:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7f2e60070bf0 msgr2=0x7f2e60073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7f2e60070bf0 0x7f2e60073000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e60068340 msgr2=0x7f2e6006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e60068340 0x7f2e6006a7f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e7406feb0 tx=0x7f2e64006cb0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e740adda0 msgr2=0x7f2e74163490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 --2- 172.21.15.116:0/2912212319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e740adda0 0x7f2e74163490 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c014040 tx=0x7f2e6c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e78d56700 1 -- 172.21.15.116:0/2912212319 reap_dead start 2023-11-08T08:18:48.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 shutdown_connections 2023-11-08T08:18:48.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 >> 172.21.15.116:0/2912212319 conn(0x7f2e740cb540 msgr2=0x7f2e740cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 shutdown_connections 2023-11-08T08:18:48.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f2e7a7b9700 1 -- 172.21.15.116:0/2912212319 wait complete. 2023-11-08T08:18:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.678+0000 7f68f57fa700 1 -- 172.21.15.116:0/3216238061 <== osd.1 v2:172.21.15.116:6800/121158 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f68f80743f0 con 0x7f68e4070bf0 2023-11-08T08:18:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.678+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f68e4070bf0 msgr2=0x7f68e4073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.678+0000 7f68ff21d700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f68e4070bf0 0x7f68e4073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.684+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68e40683c0 msgr2=0x7f68e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.684+0000 7f68ff21d700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68e40683c0 0x7f68e406a870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68f80c5610 tx=0x7f68ec009110 comp rx=0 tx=0).stop 2023-11-08T08:18:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.684+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f806d230 msgr2=0x7f68f81f2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.684+0000 7f68ff21d700 1 --2- 172.21.15.116:0/3216238061 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f68f806d230 0x7f68f81f2010 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f68e801a000 tx=0x7f68e8004750 comp rx=0 tx=0).stop 2023-11-08T08:18:48.777 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:48.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f19ec08ac20 msgr2=0x7f19ec0c7670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f19ec08ac20 0x7f19ec0c7670 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19d00682f0 msgr2=0x7f19d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19d00682f0 0x7f19d006a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0cd9d0 tx=0x7f19dc009040 comp rx=0 tx=0).stop 2023-11-08T08:18:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ec0ae020 msgr2=0x7f19ec167060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 --2- 172.21.15.116:0/4261134187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ec0ae020 0x7f19ec167060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19e800bc50 tx=0x7f19e800bf60 comp rx=0 tx=0).stop 2023-11-08T08:18:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f251b700 1 -- 172.21.15.116:0/4261134187 reap_dead start 2023-11-08T08:18:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 shutdown_connections 2023-11-08T08:18:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.677+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 >> 172.21.15.116:0/4261134187 conn(0x7f19ec06aed0 msgr2=0x7f19ec070460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 shutdown_connections 2023-11-08T08:18:48.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f19f3f7e700 1 -- 172.21.15.116:0/4261134187 wait complete. 2023-11-08T08:18:48.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-08T08:18:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.678+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 --> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f24500c7790 con 0x7f2434070bf0 2023-11-08T08:18:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7f244effd700 1 -- 172.21.15.116:0/2128227291 <== osd.5 v2:172.21.15.129:6808/121421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f24500c7790 con 0x7f2434070bf0 2023-11-08T08:18:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f2434070bf0 msgr2=0x7f2434073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f2434070bf0 0x7f2434073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2434068340 msgr2=0x7f243406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2434068340 0x7f243406a7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2444000f80 tx=0x7f2444009030 comp rx=0 tx=0).stop 2023-11-08T08:18:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24500c6530 msgr2=0x7f2450154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 --2- 172.21.15.116:0/2128227291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24500c6530 0x7f2450154fa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f24500c5f80 tx=0x7f244800fb70 comp rx=0 tx=0).stop 2023-11-08T08:18:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f24568b7700 1 -- 172.21.15.116:0/2128227291 reap_dead start 2023-11-08T08:18:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 shutdown_connections 2023-11-08T08:18:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 >> 172.21.15.116:0/2128227291 conn(0x7f24500b16c0 msgr2=0x7f2450070580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 shutdown_connections 2023-11-08T08:18:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7f245831a700 1 -- 172.21.15.116:0/2128227291 wait complete. 2023-11-08T08:18:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.676+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== osd.6 v2:172.21.15.129:6824/121459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdc481ff110 con 0x7fdc480c8b10 2023-11-08T08:18:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.679+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7040743f0 con 0x7ff70415ffa0 2023-11-08T08:18:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff6fa7fc700 1 -- 172.21.15.116:0/16915994 <== osd.7 v2:172.21.15.129:6816/121422 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff7040743f0 con 0x7ff70415ffa0 2023-11-08T08:18:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7ff70415ffa0 msgr2=0x7ff7040c3a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7ff70415ffa0 0x7ff7040c3a20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6e80682c0 msgr2=0x7ff6e806a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6e80682c0 0x7ff6e806a770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006fd0 tx=0x7ff6f4008040 comp rx=0 tx=0).stop 2023-11-08T08:18:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff704161230 msgr2=0x7ff70406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 --2- 172.21.15.116:0/16915994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff704161230 0x7ff70406a270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00b370 tx=0x7ff6fc004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70a42b700 1 -- 172.21.15.116:0/16915994 reap_dead start 2023-11-08T08:18:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 shutdown_connections 2023-11-08T08:18:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.680+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 >> 172.21.15.116:0/16915994 conn(0x7ff7040b16c0 msgr2=0x7ff704071110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 shutdown_connections 2023-11-08T08:18:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.681+0000 7ff70be8e700 1 -- 172.21.15.116:0/16915994 wait complete. 2023-11-08T08:18:48.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.684+0000 7f68fd7ba700 1 -- 172.21.15.116:0/3216238061 reap_dead start 2023-11-08T08:18:48.788 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:48.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-08T08:18:48.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.686+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 shutdown_connections 2023-11-08T08:18:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.686+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 >> 172.21.15.116:0/3216238061 conn(0x7f68f806aed0 msgr2=0x7f68f80cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.686+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 shutdown_connections 2023-11-08T08:18:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.686+0000 7f68ff21d700 1 -- 172.21.15.116:0/3216238061 wait complete. 2023-11-08T08:18:48.791 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:48.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-08T08:18:48.793 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:48.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-08T08:18:48.794 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-08T08:18:48.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-08T08:18:48.795 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:48.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-08T08:18:48.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.704+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 --> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdc48067d70 con 0x7fdc480c8b10 2023-11-08T08:18:48.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.704+0000 7fdc2ffff700 1 -- 172.21.15.116:0/3834660530 <== osd.6 v2:172.21.15.129:6824/121459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdc48067d70 con 0x7fdc480c8b10 2023-11-08T08:18:48.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7fdc480c8b10 msgr2=0x7fdc481603a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7fdc480c8b10 0x7fdc481603a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdc340682f0 msgr2=0x7fdc3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdc340682f0 0x7fdc3406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005910 tx=0x7fdc3c001150 comp rx=0 tx=0).stop 2023-11-08T08:18:48.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 msgr2=0x7fdc481fc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:48.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 --2- 172.21.15.116:0/3834660530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc4806b220 0x7fdc481fc470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000efc0 tx=0x7fdc4000c0c0 comp rx=0 tx=0).stop 2023-11-08T08:18:48.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc47fff700 1 -- 172.21.15.116:0/3834660530 reap_dead start 2023-11-08T08:18:48.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 shutdown_connections 2023-11-08T08:18:48.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 >> 172.21.15.116:0/3834660530 conn(0x7fdc4806c520 msgr2=0x7fdc480b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:48.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 shutdown_connections 2023-11-08T08:18:48.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:48.705+0000 7fdc4df1f700 1 -- 172.21.15.116:0/3834660530 wait complete. 2023-11-08T08:18:48.800 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:48.801 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-08T08:18:48.802 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:48.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-08T08:18:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.034+0000 7f6519e9d700 1 Processor -- start 2023-11-08T08:18:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.034+0000 7f6519e9d700 1 -- start start 2023-11-08T08:18:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f6519e9d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65141609c0 0x7f65140ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f6519e9d700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65140cc690 con 0x7f65140b0c50 2023-11-08T08:18:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f6519e9d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65140cc8c0 con 0x7f651406cdd0 2023-11-08T08:18:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f6519e9d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65140caa30 con 0x7f65141609c0 2023-11-08T08:18:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f6513fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65140b0c50 0x7f651406bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34322/0 (socket says 172.21.15.116:34322) 2023-11-08T08:18:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f6513fff700 1 -- 172.21.15.116:0/44326709 learned_addr learned my addr 172.21.15.116:0/44326709 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.036+0000 7f65137fe700 1 --2- 172.21.15.116:0/44326709 >> v2:172.21.15.129:3300/0 conn(0x7f65141609c0 0x7f65140ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085617009 0 0) 0x7f65140cc690 con 0x7f65140b0c50 2023-11-08T08:18:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64fc0035b0 con 0x7f65140b0c50 2023-11-08T08:18:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490840133 0 0) 0x7f65140cc8c0 con 0x7f651406cdd0 2023-11-08T08:18:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65140cc690 con 0x7f651406cdd0 2023-11-08T08:18:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795055167 0 0) 0x7f64fc0035b0 con 0x7f65140b0c50 2023-11-08T08:18:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 >> v2:172.21.15.129:3300/0 conn(0x7f65141609c0 msgr2=0x7f65140ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 --2- 172.21.15.116:0/44326709 >> v2:172.21.15.129:3300/0 conn(0x7f65141609c0 0x7f65140ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 >> v1:172.21.15.129:6789/0 conn(0x7f651406cdd0 legacy=0x7f65140b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65140cad10 con 0x7f65140b0c50 2023-11-08T08:18:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f6504001670 con 0x7f65140b0c50 2023-11-08T08:18:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 >> v1:172.21.15.116:6789/0 conn(0x7f65140b0c50 legacy=0x7f651406bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64fc003b90 0x7f64fc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f64fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64fc005200 0x7f64fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65140caa30 con 0x7f64fc005200 2023-11-08T08:18:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00c110 con 0x7f64fc004850 2023-11-08T08:18:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00c2d0 con 0x7f64fc003b90 2023-11-08T08:18:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.037+0000 7f81b9195700 1 Processor -- start 2023-11-08T08:18:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.038+0000 7f81b9195700 1 -- start start 2023-11-08T08:18:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.038+0000 7f65137fe700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64fc003b90 0x7f64fc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f6512ffd700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f64fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b9195700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 0x7f81b41651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b9195700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f81b4161b40 0x7f81b40b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b9195700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b4165720 con 0x7f81b40b2d10 2023-11-08T08:18:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b9195700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40b3150 con 0x7f81b4161b40 2023-11-08T08:18:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b9195700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40cfc40 con 0x7f81b4164e10 2023-11-08T08:18:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 0x7f81b41651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 0x7f81b41651e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52372/0 (socket says 172.21.15.116:52372) 2023-11-08T08:18:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 -- 172.21.15.116:0/1202076499 learned_addr learned my addr 172.21.15.116:0/1202076499 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 -- 172.21.15.116:0/1202076499 >> v2:172.21.15.129:3300/0 conn(0x7f81b4161b40 msgr2=0x7f81b40b27d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:18:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 --2- 172.21.15.116:0/1202076499 >> v2:172.21.15.129:3300/0 conn(0x7f81b4161b40 0x7f81b40b27d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 -- 172.21.15.116:0/1202076499 >> v1:172.21.15.129:6789/0 conn(0x7f81b40b2d10 legacy=0x7f81b40d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f81b2d9d700 1 -- 172.21.15.116:0/1202076499 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b406c3c0 con 0x7f81b4164e10 2023-11-08T08:18:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f6512ffd700 1 -- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64fc003b90 msgr2=0x7f64fc003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f6512ffd700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f64fc003b90 0x7f64fc003f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f6512ffd700 1 -- 172.21.15.116:0/44326709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64fc005200 msgr2=0x7f64fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f6512ffd700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64fc005200 0x7f64fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.039+0000 7f6512ffd700 1 -- 172.21.15.116:0/44326709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65140cad10 con 0x7f64fc004850 2023-11-08T08:18:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f81b2d9d700 1 --2- 172.21.15.116:0/1202076499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 0x7f81b41651e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81a80098f0 tx=0x7f81a8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37eaafd441d14d95 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f6512ffd700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f64fc004cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f650c002230 tx=0x7f64fc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43503310ff51c18f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f6513fff700 1 -- 172.21.15.116:0/44326709 reap_dead start 2023-11-08T08:18:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c001f30 con 0x7f64fc004850 2023-11-08T08:18:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c00ce60 con 0x7f64fc004850 2023-11-08T08:18:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f65127fc700 1 -- 172.21.15.116:0/44326709 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c003c20 con 0x7f64fc004850 2023-11-08T08:18:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f81b1d9b700 1 -- 172.21.15.116:0/1202076499 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8004230 con 0x7f81b4164e10 2023-11-08T08:18:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f81b1d9b700 1 -- 172.21.15.116:0/1202076499 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a8004390 con 0x7f81b4164e10 2023-11-08T08:18:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f81b1d9b700 1 -- 172.21.15.116:0/1202076499 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80036a0 con 0x7f81b4164e10 2023-11-08T08:18:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f6519e9d700 1 -- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 msgr2=0x7f64fc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f6519e9d700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f64fc004cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f650c002230 tx=0x7f64fc005880 comp rx=0 tx=0).stop 2023-11-08T08:18:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 -- 172.21.15.116:0/1202076499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 msgr2=0x7f81b41651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 --2- 172.21.15.116:0/1202076499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 0x7f81b41651e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81a80098f0 tx=0x7f81a8009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f6519e9d700 1 -- 172.21.15.116:0/44326709 shutdown_connections 2023-11-08T08:18:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f6519e9d700 1 --2- 172.21.15.116:0/44326709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f64fc004cc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f6519e9d700 1 -- 172.21.15.116:0/44326709 >> 172.21.15.116:0/44326709 conn(0x7f651406aed0 msgr2=0x7f65140cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 -- 172.21.15.116:0/1202076499 shutdown_connections 2023-11-08T08:18:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 --2- 172.21.15.116:0/1202076499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4164e10 0x7f81b41651e0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81a80098f0 tx=0x7f81a8009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 --2- 172.21.15.116:0/1202076499 >> v2:172.21.15.129:3300/0 conn(0x7f81b4161b40 0x7f81b40b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 -- 172.21.15.116:0/1202076499 >> 172.21.15.116:0/1202076499 conn(0x7f81b406ea80 msgr2=0x7f81b406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 -- 172.21.15.116:0/1202076499 shutdown_connections 2023-11-08T08:18:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f81b9195700 1 -- 172.21.15.116:0/1202076499 wait complete. 2023-11-08T08:18:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.040+0000 7f47deeac700 1 Processor -- start 2023-11-08T08:18:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.041+0000 7f47deeac700 1 -- start start 2023-11-08T08:18:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47deeac700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 0x7f47d806c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47deeac700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47d808ac20 0x7f47d80cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47deeac700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d80cb160 con 0x7f47d806c930 2023-11-08T08:18:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47deeac700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d806b910 con 0x7f47d808ac20 2023-11-08T08:18:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47deeac700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d806bbf0 con 0x7f47d80ae020 2023-11-08T08:18:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47dcc48700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 0x7f47d806c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47d7fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47d808ac20 0x7f47d80cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47dcc48700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 0x7f47d806c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52376/0 (socket says 172.21.15.116:52376) 2023-11-08T08:18:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.042+0000 7f47dcc48700 1 -- 172.21.15.116:0/3516410636 learned_addr learned my addr 172.21.15.116:0/3516410636 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f81b9195700 1 Processor -- start 2023-11-08T08:18:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f47dcc48700 1 -- 172.21.15.116:0/3516410636 >> v2:172.21.15.129:3300/0 conn(0x7f47d808ac20 msgr2=0x7f47d80cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f47dcc48700 1 --2- 172.21.15.116:0/3516410636 >> v2:172.21.15.129:3300/0 conn(0x7f47d808ac20 0x7f47d80cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f47dcc48700 1 -- 172.21.15.116:0/3516410636 >> v1:172.21.15.129:6789/0 conn(0x7f47d806c930 legacy=0x7f47d80b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f47dcc48700 1 -- 172.21.15.116:0/3516410636 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d806bed0 con 0x7f47d80ae020 2023-11-08T08:18:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f47dcc48700 1 --2- 172.21.15.116:0/3516410636 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 0x7f47d806c3f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0082f0 tx=0x7f47cc00ad80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8753fdfd1d0941 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7fcbf0036700 1 Processor -- start 2023-11-08T08:18:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.043+0000 7f81b9195700 1 -- start start 2023-11-08T08:18:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b9195700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b40b2d10 0x7f81b4160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b9195700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4161b40 0x7f81b40c1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b9195700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 0x7f81b40bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b9195700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4160bb0 con 0x7f81b40b2d10 2023-11-08T08:18:49.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b9195700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40c01f0 con 0x7f81b40c6800 2023-11-08T08:18:49.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b9195700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40c04d0 con 0x7f81b4161b40 2023-11-08T08:18:49.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 0x7f81b40bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 0x7f81b40bf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57186/0 (socket says 172.21.15.116:57186) 2023-11-08T08:18:49.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 -- 172.21.15.116:0/3380927863 learned_addr learned my addr 172.21.15.116:0/3380927863 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 -- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4161b40 msgr2=0x7f81b40c1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4161b40 0x7f81b40c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 -- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b40b2d10 msgr2=0x7f81b4160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7f81b359e700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b40b2d10 0x7f81b4160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7f81b359e700 1 -- 172.21.15.116:0/3380927863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8009550 con 0x7f81b40c6800 2023-11-08T08:18:49.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7f81b359e700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 0x7f81b40bf8f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00b990 tx=0x7f81ac00bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7fe5fb65c700 1 Processor -- start 2023-11-08T08:18:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7f6519e9d700 1 -- 172.21.15.116:0/44326709 shutdown_connections 2023-11-08T08:18:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.044+0000 7fcbf0036700 1 -- start start 2023-11-08T08:18:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7fcbf0036700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcbe806b220 0x7fcbe80cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7fcbf0036700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80d22c0 con 0x7fcbe81652c0 2023-11-08T08:18:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7fcbf0036700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80cf550 con 0x7fcbe8160ac0 2023-11-08T08:18:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7fcbf0036700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe80d0840 con 0x7fcbe806b220 2023-11-08T08:18:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7fcbee5d3700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcbe81652c0 0x7fcbe80d0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34332/0 (socket says 172.21.15.116:34332) 2023-11-08T08:18:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7fcbee5d3700 1 -- 172.21.15.116:0/2880411373 learned_addr learned my addr 172.21.15.116:0/2880411373 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183517194 0 0) 0x7fcbe80d22c0 con 0x7fcbe81652c0 2023-11-08T08:18:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd40035b0 con 0x7fcbe81652c0 2023-11-08T08:18:49.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143511032 0 0) 0x7fcbd40035b0 con 0x7fcbe81652c0 2023-11-08T08:18:49.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 >> v2:172.21.15.129:3300/0 conn(0x7fcbe806b220 msgr2=0x7fcbe80cef20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:18:49.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 --2- 172.21.15.116:0/2880411373 >> v2:172.21.15.129:3300/0 conn(0x7fcbe806b220 0x7fcbe80cef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fe5fb65c700 1 -- start start 2023-11-08T08:18:49.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 >> v1:172.21.15.129:6789/0 conn(0x7fcbe8160ac0 legacy=0x7fcbe806ab10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-08T08:18:49.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe81647e0 con 0x7fcbe81652c0 2023-11-08T08:18:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fcbe00019f0 con 0x7fcbe81652c0 2023-11-08T08:18:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 >> v1:172.21.15.116:6789/0 conn(0x7fcbe81652c0 legacy=0x7fcbe80d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbd4003a00 0x7fcbd4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4004850 0x7fcbd4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd4005320 0x7fcbd400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd400c1b0 con 0x7fcbd4005320 2023-11-08T08:18:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd400c320 con 0x7fcbd4004850 2023-11-08T08:18:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd400c4e0 con 0x7fcbd4003a00 2023-11-08T08:18:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5fb65c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 0x7fe5f4160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5fb65c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe5f40b5dc0 0x7fe5f406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5fb65c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f40d2d30 con 0x7fe5f406b050 2023-11-08T08:18:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5fb65c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f406b4d0 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5fb65c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f40d0b40 con 0x7fe5f415c8c0 2023-11-08T08:18:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 0x7fe5f4160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 0x7fe5f4160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52400/0 (socket says 172.21.15.116:52400) 2023-11-08T08:18:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 -- 172.21.15.116:0/122775936 learned_addr learned my addr 172.21.15.116:0/122775936 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f8bf7700 1 --2- 172.21.15.116:0/122775936 >> v2:172.21.15.129:3300/0 conn(0x7fe5f40b5dc0 0x7fe5f406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 -- 172.21.15.116:0/122775936 >> v2:172.21.15.129:3300/0 conn(0x7fe5f40b5dc0 msgr2=0x7fe5f406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 --2- 172.21.15.116:0/122775936 >> v2:172.21.15.129:3300/0 conn(0x7fe5f40b5dc0 0x7fe5f406ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 -- 172.21.15.116:0/122775936 >> v1:172.21.15.116:6789/0 conn(0x7fe5f406b050 legacy=0x7fe5f40d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 -- 172.21.15.116:0/122775936 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f41611f0 con 0x7fe5f415c8c0 2023-11-08T08:18:49.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f8bf7700 1 --2- 172.21.15.116:0/122775936 >> v2:172.21.15.129:3300/0 conn(0x7fe5f40b5dc0 0x7fe5f406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:49.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fe5f93f8700 1 --2- 172.21.15.116:0/122775936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 0x7fe5f4160cb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00098f0 tx=0x7fe5f0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e1e65a1e2cd0007 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fe5ebfff700 1 -- 172.21.15.116:0/122775936 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0014e90 con 0x7fe5f415c8c0 2023-11-08T08:18:49.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fe5ebfff700 1 -- 172.21.15.116:0/122775936 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0004030 con 0x7fe5f415c8c0 2023-11-08T08:18:49.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fe5ebfff700 1 -- 172.21.15.116:0/122775936 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00043c0 con 0x7fe5f415c8c0 2023-11-08T08:18:49.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbd4003a00 0x7fcbd4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fcbee5d3700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd4005320 0x7fcbd400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbeddd2700 1 -- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4004850 msgr2=0x7fcbd4004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-08T08:18:49.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4004850 0x7fcbd4004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbeddd2700 1 -- 172.21.15.116:0/2880411373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd4005320 msgr2=0x7fcbd400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd4005320 0x7fcbd400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbeddd2700 1 -- 172.21.15.116:0/2880411373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe81647e0 con 0x7fcbd4003a00 2023-11-08T08:18:49.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbee5d3700 1 -- 172.21.15.116:0/2880411373 reap_dead start 2023-11-08T08:18:49.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbd4003a00 0x7fcbd4003df0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009860 tx=0x7fcbd4005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131e6876163fae6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80092d0 con 0x7fcbd4003a00 2023-11-08T08:18:49.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8011620 con 0x7fcbd4003a00 2023-11-08T08:18:49.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fcbecdd0700 1 -- 172.21.15.116:0/2880411373 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80119d0 con 0x7fcbd4003a00 2023-11-08T08:18:49.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.045+0000 7f6519e9d700 1 -- 172.21.15.116:0/44326709 wait complete. 2023-11-08T08:18:49.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7f6519e9d700 1 Processor -- start 2023-11-08T08:18:49.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00c9d0 con 0x7f81b40c6800 2023-11-08T08:18:49.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac00cb30 con 0x7f81b40c6800 2023-11-08T08:18:49.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac014440 con 0x7f81b40c6800 2023-11-08T08:18:49.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.046+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b406c660 con 0x7f81b40c6800 2023-11-08T08:18:49.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b40cfea0 con 0x7f81b40c6800 2023-11-08T08:18:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b406c940 con 0x7f81b40c6800 2023-11-08T08:18:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fa22e382700 1 Processor -- start 2023-11-08T08:18:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7fa22e382700 1 -- start start 2023-11-08T08:18:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fe5fb65c700 1 -- 172.21.15.116:0/122775936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 msgr2=0x7fe5f4160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/122775936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 0x7fe5f4160cb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00098f0 tx=0x7fe5f0009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.047+0000 7f6519e9d700 1 -- start start 2023-11-08T08:18:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6519e9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f651415e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6519e9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 0x7f65140c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6519e9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651415f5d0 0x7f651415fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6519e9d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651415e720 con 0x7f651415f5d0 2023-11-08T08:18:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6519e9d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651415ff80 con 0x7f64fc004850 2023-11-08T08:18:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6519e9d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65140c50c0 con 0x7f65140c25d0 2023-11-08T08:18:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f65137fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f651415e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6512ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 0x7f65140c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6512ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 0x7f65140c2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52414/0 (socket says 172.21.15.116:52414) 2023-11-08T08:18:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f6512ffd700 1 -- 172.21.15.116:0/1507874002 learned_addr learned my addr 172.21.15.116:0/1507874002 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fe5fb65c700 1 -- 172.21.15.116:0/122775936 shutdown_connections 2023-11-08T08:18:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/122775936 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f415c8c0 0x7fe5f4160cb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/122775936 >> v2:172.21.15.129:3300/0 conn(0x7fe5f40b5dc0 0x7fe5f406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fe5fb65c700 1 -- 172.21.15.116:0/122775936 >> 172.21.15.116:0/122775936 conn(0x7fe5f406c520 msgr2=0x7fe5f406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.048+0000 7f47d77fe700 1 -- 172.21.15.116:0/3516410636 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc00bcf0 con 0x7f47d80ae020 2023-11-08T08:18:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f47d77fe700 1 -- 172.21.15.116:0/3516410636 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc00be50 con 0x7f47d80ae020 2023-11-08T08:18:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f47d77fe700 1 -- 172.21.15.116:0/3516410636 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc015640 con 0x7f47d80ae020 2023-11-08T08:18:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6513fff700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651415f5d0 0x7f651415fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6512ffd700 1 -- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 msgr2=0x7f651415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6512ffd700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f651415e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6512ffd700 1 -- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651415f5d0 msgr2=0x7f651415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6512ffd700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651415f5d0 0x7f651415fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6512ffd700 1 -- 172.21.15.116:0/1507874002 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65140cb2d0 con 0x7f65140c25d0 2023-11-08T08:18:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 -- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbd4003a00 msgr2=0x7fcbd4003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbd4003a00 0x7fcbd4003df0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009860 tx=0x7fcbd4005880 comp rx=0 tx=0).stop 2023-11-08T08:18:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6512ffd700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 0x7f65140c2a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00c2a0 tx=0x7f650c001650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fe5fb65c700 1 -- 172.21.15.116:0/122775936 shutdown_connections 2023-11-08T08:18:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fe5fb65c700 1 -- 172.21.15.116:0/122775936 wait complete. 2023-11-08T08:18:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 -- 172.21.15.116:0/2880411373 shutdown_connections 2023-11-08T08:18:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2880411373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbd4003a00 0x7fcbd4003df0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 -- 172.21.15.116:0/2880411373 >> 172.21.15.116:0/2880411373 conn(0x7fcbe806c520 msgr2=0x7fcbe806c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00cc60 con 0x7f65140c25d0 2023-11-08T08:18:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c0050d0 con 0x7f65140c25d0 2023-11-08T08:18:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c005480 con 0x7f65140c25d0 2023-11-08T08:18:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7fe5fb65c700 1 Processor -- start 2023-11-08T08:18:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7fe5fb65c700 1 -- start start 2023-11-08T08:18:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5fb65c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5f406b050 0x7fe5f40c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5fb65c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 0x7fe5f40c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5fb65c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f415c8c0 0x7fe5f41545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5fb65c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f40c5b90 con 0x7fe5f415c8c0 2023-11-08T08:18:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5fb65c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f4147800 con 0x7fe5f406b050 2023-11-08T08:18:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5fb65c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f4146bf0 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 -- 172.21.15.116:0/2880411373 shutdown_connections 2023-11-08T08:18:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7fcbf0036700 1 -- 172.21.15.116:0/2880411373 wait complete. 2023-11-08T08:18:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fcbf0036700 1 Processor -- start 2023-11-08T08:18:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.049+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65140caff0 con 0x7f65140c25d0 2023-11-08T08:18:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651406b810 con 0x7f65140c25d0 2023-11-08T08:18:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65140c5450 con 0x7f65140c25d0 2023-11-08T08:18:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81ac01b030 con 0x7f81b40c6800 2023-11-08T08:18:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7f819b7fe700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81a0068340 0x7f81a006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f81ac085fc0 con 0x7f81b40c6800 2023-11-08T08:18:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7f81b2d9d700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81a0068340 0x7f81a006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7f81b2d9d700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81a0068340 0x7f81a006a7f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f81b4070890 tx=0x7f81a8010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7f47deeac700 1 -- 172.21.15.116:0/3516410636 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 msgr2=0x7f47d806c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.050+0000 7f47deeac700 1 --2- 172.21.15.116:0/3516410636 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 0x7f47d806c3f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0082f0 tx=0x7f47cc00ad80 comp rx=0 tx=0).stop 2023-11-08T08:18:49.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5f8bf7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 0x7fe5f40c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5f8bf7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 0x7fe5f40c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52426/0 (socket says 172.21.15.116:52426) 2023-11-08T08:18:49.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5f8bf7700 1 -- 172.21.15.116:0/2459824090 learned_addr learned my addr 172.21.15.116:0/2459824090 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5f9bf9700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f415c8c0 0x7fe5f41545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.051+0000 7fe5f93f8700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5f406b050 0x7fe5f40c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5f8bf7700 1 -- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5f406b050 msgr2=0x7fe5f40c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5f8bf7700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5f406b050 0x7fe5f40c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5f8bf7700 1 -- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f415c8c0 msgr2=0x7fe5f41545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5f8bf7700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f415c8c0 0x7fe5f41545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5f8bf7700 1 -- 172.21.15.116:0/2459824090 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec009a70 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5f8bf7700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 0x7fe5f40c6ea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009f60 tx=0x7fe5ec009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec00cdc0 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec00f040 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0146a0 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0009550 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f40cd760 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7fcbf0036700 1 -- start start 2023-11-08T08:18:49.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbf0036700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4003a00 0x7fcbe81f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbf0036700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 0x7fcbe8151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbf0036700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 0x7fcbe80c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbf0036700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81f7480 con 0x7fcbe80c6aa0 2023-11-08T08:18:49.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbf0036700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81f6be0 con 0x7fcbd4003a00 2023-11-08T08:18:49.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbf0036700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81f6ec0 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 0x7fcbe8151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbee5d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 0x7fcbe80c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 0x7fcbe8151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52434/0 (socket says 172.21.15.116:52434) 2023-11-08T08:18:49.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 -- 172.21.15.116:0/2610960444 learned_addr learned my addr 172.21.15.116:0/2610960444 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbee5d3700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 0x7fcbe80c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39272/0 (socket says 172.21.15.116:39272) 2023-11-08T08:18:49.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4003a00 0x7fcbe81f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.052+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f650c02d070 con 0x7f65140c25d0 2023-11-08T08:18:49.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7f64fbfff700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65000682f0 0x7f650006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f650c0870b0 con 0x7f65140c25d0 2023-11-08T08:18:49.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7f65137fe700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65000682f0 0x7f650006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 -- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4003a00 msgr2=0x7fcbe81f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4003a00 0x7fcbe81f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 -- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 msgr2=0x7fcbe80c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 0x7fcbe80c6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbed5d1700 1 -- 172.21.15.116:0/2610960444 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe8164da0 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7f65137fe700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65000682f0 0x7f650006a7a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f651415f360 tx=0x7f6508008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.053+0000 7fcbee5d3700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 0x7fcbe80c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:49.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fcbed5d1700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 0x7fcbe8151610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b3c0 tx=0x7fcbe400b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f47deeac700 1 -- 172.21.15.116:0/3516410636 shutdown_connections 2023-11-08T08:18:49.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f47deeac700 1 --2- 172.21.15.116:0/3516410636 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d80ae020 0x7f47d806c3f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f47deeac700 1 --2- 172.21.15.116:0/3516410636 >> v2:172.21.15.129:3300/0 conn(0x7f47d808ac20 0x7f47d80cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f47deeac700 1 -- 172.21.15.116:0/3516410636 >> 172.21.15.116:0/3516410636 conn(0x7f47d806aed0 msgr2=0x7f47d80cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fa22e382700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2281609c0 0x7fa2280ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f04e3691700 1 Processor -- start 2023-11-08T08:18:49.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fa22e382700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2280cc690 con 0x7fa22806cdd0 2023-11-08T08:18:49.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fa22e382700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2280cc8c0 con 0x7fa2280b0c50 2023-11-08T08:18:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fa22e382700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2280caa30 con 0x7fa2281609c0 2023-11-08T08:18:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f47deeac700 1 -- 172.21.15.116:0/3516410636 shutdown_connections 2023-11-08T08:18:49.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f40d2a50 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fa22c91f700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa2280b0c50 0x7fa22806bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57306/0 (socket says 172.21.15.116:57306) 2023-11-08T08:18:49.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7fa22c91f700 1 -- 172.21.15.116:0/3085872165 learned_addr learned my addr 172.21.15.116:0/3085872165 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:49.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.054+0000 7f04e3691700 1 -- start start 2023-11-08T08:18:49.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f6feaeeb700 1 Processor -- start 2023-11-08T08:18:49.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f6feaeeb700 1 -- start start 2023-11-08T08:18:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e3691700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc06d230 0x7f04dc0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e3691700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04dc0b0c50 0x7f04dc0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e3691700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc06d6c0 con 0x7f04dc158980 2023-11-08T08:18:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e3691700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc06c700 con 0x7f04dc0b0c50 2023-11-08T08:18:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e3691700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0ca7f0 con 0x7f04dc06d230 2023-11-08T08:18:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e142d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f04dc158980 0x7f04dc0b0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34346/0 (socket says 172.21.15.116:34346) 2023-11-08T08:18:49.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e142d700 1 -- 172.21.15.116:0/767036646 learned_addr learned my addr 172.21.15.116:0/767036646 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7f04e0c2c700 1 --2- 172.21.15.116:0/767036646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc06d230 0x7f04dc0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f04e1c2e700 1 --2- 172.21.15.116:0/767036646 >> v2:172.21.15.129:3300/0 conn(0x7f04dc0b0c50 0x7f04dc0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f04e0c2c700 1 -- 172.21.15.116:0/767036646 >> v2:172.21.15.129:3300/0 conn(0x7f04dc0b0c50 msgr2=0x7f04dc0ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f04e0c2c700 1 --2- 172.21.15.116:0/767036646 >> v2:172.21.15.129:3300/0 conn(0x7f04dc0b0c50 0x7f04dc0ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f04e0c2c700 1 -- 172.21.15.116:0/767036646 >> v1:172.21.15.116:6789/0 conn(0x7f04dc158980 legacy=0x7f04dc0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f04e0c2c700 1 -- 172.21.15.116:0/767036646 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0caad0 con 0x7f04dc06d230 2023-11-08T08:18:49.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6feaeeb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6fe40b50d0 0x7f6fe406b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6feaeeb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe415a9a0 0x7f6fe406a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6feaeeb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40cba60 con 0x7f6fe415a500 2023-11-08T08:18:49.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6feaeeb700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe40cd480 con 0x7f6fe40b50d0 2023-11-08T08:18:49.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6feaeeb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40cd760 con 0x7f6fe415a9a0 2023-11-08T08:18:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe3fff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6fe415a500 0x7f6fe406c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57320/0 (socket says 172.21.15.116:57320) 2023-11-08T08:18:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe3fff700 1 -- 172.21.15.116:0/708787257 learned_addr learned my addr 172.21.15.116:0/708787257 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe9488700 1 --2- 172.21.15.116:0/708787257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe415a9a0 0x7f6fe406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81ac052970 con 0x7f81b40c6800 2023-11-08T08:18:49.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f04e0c2c700 1 --2- 172.21.15.116:0/767036646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc06d230 0x7f04dc0adf60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009840 tx=0x7f04cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e202c3b9bef83a16 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7fa227fff700 1 --2- 172.21.15.116:0/3085872165 >> v2:172.21.15.116:3300/0 conn(0x7fa2281609c0 0x7fa2280ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7fa227fff700 1 -- 172.21.15.116:0/3085872165 >> v1:172.21.15.129:6789/0 conn(0x7fa2280b0c50 legacy=0x7fa22806bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7fa227fff700 1 -- 172.21.15.116:0/3085872165 >> v1:172.21.15.116:6789/0 conn(0x7fa22806cdd0 legacy=0x7fa2280b0540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-08T08:18:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7fa227fff700 1 -- 172.21.15.116:0/3085872165 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2280cad10 con 0x7fa2281609c0 2023-11-08T08:18:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.055+0000 7fa227fff700 1 --2- 172.21.15.116:0/3085872165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2281609c0 0x7fa2280ca150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa218009840 tx=0x7fa218009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8075c5b5663914a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe8c87700 1 --2- 172.21.15.116:0/708787257 >> v2:172.21.15.116:3300/0 conn(0x7f6fe40b50d0 0x7f6fe406b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe37fe700 1 -- 172.21.15.116:0/708787257 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245905611 0 0) 0x7f6fe40cba60 con 0x7f6fe415a500 2023-11-08T08:18:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe37fe700 1 -- 172.21.15.116:0/708787257 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd40035b0 con 0x7f6fe415a500 2023-11-08T08:18:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe37fe700 1 -- 172.21.15.116:0/708787257 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1676656057 0 0) 0x7f6fd40035b0 con 0x7f6fe415a500 2023-11-08T08:18:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f6fe8c87700 1 -- 172.21.15.116:0/708787257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe415a9a0 msgr2=0x7f6fe406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe8c87700 1 --2- 172.21.15.116:0/708787257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe415a9a0 0x7f6fe406a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.056+0000 7f47deeac700 1 -- 172.21.15.116:0/3516410636 wait complete. 2023-11-08T08:18:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe8c87700 1 -- 172.21.15.116:0/708787257 >> v1:172.21.15.129:6789/0 conn(0x7f6fe415a500 legacy=0x7f6fe406c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe8c87700 1 -- 172.21.15.116:0/708787257 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40cda40 con 0x7f6fe40b50d0 2023-11-08T08:18:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7fa226ffd700 1 -- 172.21.15.116:0/3085872165 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218004230 con 0x7fa2281609c0 2023-11-08T08:18:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7fa226ffd700 1 -- 172.21.15.116:0/3085872165 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa218004390 con 0x7fa2281609c0 2023-11-08T08:18:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7fa226ffd700 1 -- 172.21.15.116:0/3085872165 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180036a0 con 0x7fa2281609c0 2023-11-08T08:18:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe8c87700 1 --2- 172.21.15.116:0/708787257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe40b50d0 0x7f6fe406b5b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009840 tx=0x7f6fd8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69749470a3aa2e6f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe37fe700 1 -- 172.21.15.116:0/708787257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0001670 con 0x7f6fe40b50d0 2023-11-08T08:18:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe37fe700 1 -- 172.21.15.116:0/708787257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd8004030 con 0x7f6fe40b50d0 2023-11-08T08:18:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7f6fe37fe700 1 -- 172.21.15.116:0/708787257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd80043c0 con 0x7f6fe40b50d0 2023-11-08T08:18:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7fa22e382700 1 -- 172.21.15.116:0/3085872165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2281609c0 msgr2=0x7fa2280ca150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.057+0000 7fa22e382700 1 --2- 172.21.15.116:0/3085872165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2281609c0 0x7fa2280ca150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa218009840 tx=0x7fa218009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:49.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7f47deeac700 1 Processor -- start 2023-11-08T08:18:49.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7f6feaeeb700 1 -- 172.21.15.116:0/708787257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe40b50d0 msgr2=0x7f6fe406b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/708787257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe40b50d0 0x7f6fe406b5b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009840 tx=0x7f6fd8009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:49.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7fa22e382700 1 -- 172.21.15.116:0/3085872165 shutdown_connections 2023-11-08T08:18:49.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7fa22e382700 1 --2- 172.21.15.116:0/3085872165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2281609c0 0x7fa2280ca150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7fa22e382700 1 -- 172.21.15.116:0/3085872165 >> 172.21.15.116:0/3085872165 conn(0x7fa22806aed0 msgr2=0x7fa2280cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7fa22e382700 1 -- 172.21.15.116:0/3085872165 shutdown_connections 2023-11-08T08:18:49.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7fa22e382700 1 -- 172.21.15.116:0/3085872165 wait complete. 2023-11-08T08:18:49.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.058+0000 7f47deeac700 1 -- start start 2023-11-08T08:18:49.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47deeac700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d806c930 0x7f47d80c2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47deeac700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 0x7f47d8206cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47deeac700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d8207200 0x7f47d80cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47deeac700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80c4ff0 con 0x7f47d80ae020 2023-11-08T08:18:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47deeac700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80c4140 con 0x7f47d806c930 2023-11-08T08:18:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47deeac700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80c43e0 con 0x7f47d8207200 2023-11-08T08:18:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 0x7f47d8206cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 0x7f47d8206cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39298/0 (socket says 172.21.15.116:39298) 2023-11-08T08:18:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 -- 172.21.15.116:0/2734097085 learned_addr learned my addr 172.21.15.116:0/2734097085 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f6feaeeb700 1 -- 172.21.15.116:0/708787257 shutdown_connections 2023-11-08T08:18:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/708787257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe40b50d0 0x7f6fe406b5b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/708787257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe415a9a0 0x7f6fe406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f6feaeeb700 1 -- 172.21.15.116:0/708787257 >> 172.21.15.116:0/708787257 conn(0x7f6fe40cf400 msgr2=0x7f6fe40cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f6feaeeb700 1 -- 172.21.15.116:0/708787257 shutdown_connections 2023-11-08T08:18:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 -- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d8207200 msgr2=0x7f47d80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47dd449700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d8207200 0x7f47d80cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47dcc48700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d806c930 0x7f47d80c2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7fa22e382700 1 Processor -- start 2023-11-08T08:18:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d8207200 0x7f47d80cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 -- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d806c930 msgr2=0x7f47d80c2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d806c930 0x7f47d80c2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47d7fff700 1 -- 172.21.15.116:0/2734097085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc00a700 con 0x7f47d80ae020 2023-11-08T08:18:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f47dcc48700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d806c930 0x7f47d80c2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.060+0000 7f47dd449700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d8207200 0x7f47d80cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.060+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f650c053820 con 0x7f65140c25d0 2023-11-08T08:18:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.060+0000 7f04d3fff700 1 -- 172.21.15.116:0/767036646 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc011830 con 0x7f04dc06d230 2023-11-08T08:18:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.060+0000 7f04d3fff700 1 -- 172.21.15.116:0/767036646 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc011990 con 0x7f04dc06d230 2023-11-08T08:18:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.060+0000 7f04d3fff700 1 -- 172.21.15.116:0/767036646 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc011d20 con 0x7f04dc06d230 2023-11-08T08:18:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.060+0000 7fa22e382700 1 -- start start 2023-11-08T08:18:49.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.059+0000 7f6feaeeb700 1 -- 172.21.15.116:0/708787257 wait complete. 2023-11-08T08:18:49.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f6feaeeb700 1 Processor -- start 2023-11-08T08:18:49.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22e382700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 0x7fa2280c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22e382700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2280b0c50 0x7fa2280afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22e382700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2281609c0 0x7fa2281f9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22e382700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228162770 con 0x7fa2280b0c50 2023-11-08T08:18:49.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22e382700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228162980 con 0x7fa22806cdd0 2023-11-08T08:18:49.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22e382700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa228162c60 con 0x7fa2281609c0 2023-11-08T08:18:49.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f04e3691700 1 -- 172.21.15.116:0/767036646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc06d230 msgr2=0x7f04dc0adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f04e3691700 1 --2- 172.21.15.116:0/767036646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc06d230 0x7f04dc0adf60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009840 tx=0x7f04cc009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:49.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f04e3691700 1 -- 172.21.15.116:0/767036646 shutdown_connections 2023-11-08T08:18:49.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f04e3691700 1 --2- 172.21.15.116:0/767036646 >> v2:172.21.15.129:3300/0 conn(0x7f04dc0b0c50 0x7f04dc0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f04e3691700 1 --2- 172.21.15.116:0/767036646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc06d230 0x7f04dc0adf60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f04e3691700 1 -- 172.21.15.116:0/767036646 >> 172.21.15.116:0/767036646 conn(0x7f04dc06aed0 msgr2=0x7f04dc0cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 0x7fa2280c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 0x7fa2280c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57262/0 (socket says 172.21.15.116:57262) 2023-11-08T08:18:49.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 -- 172.21.15.116:0/2048436995 learned_addr learned my addr 172.21.15.116:0/2048436995 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa22c91f700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2281609c0 0x7fa2281f9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 -- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2281609c0 msgr2=0x7fa2281f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2281609c0 0x7fa2281f9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 -- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2280b0c50 msgr2=0x7fa2280afac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-08T08:18:49.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2280b0c50 0x7fa2280afac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fa227fff700 1 -- 172.21.15.116:0/2048436995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180094a0 con 0x7fa22806cdd0 2023-11-08T08:18:49.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fa227fff700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 0x7fa2280c2fe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa22806f8e0 tx=0x7fa218004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4011620 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5ec014800 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fe5e9ffb700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5d80682f0 0x7fe5d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe5ec087130 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.061+0000 7f6feaeeb700 1 -- start start 2023-11-08T08:18:49.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6feaeeb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe40b50d0 0x7f6fe40c2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6feaeeb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fe415a500 0x7f6fe406a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6feaeeb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe414b750 0x7f6fe414bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6feaeeb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe415c9c0 con 0x7f6fe414b750 2023-11-08T08:18:49.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6feaeeb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4159cb0 con 0x7f6fe415a500 2023-11-08T08:18:49.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6feaeeb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe414c540 con 0x7f6fe40b50d0 2023-11-08T08:18:49.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6fe8c87700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe40b50d0 0x7f6fe40c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6fe9488700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe414b750 0x7f6fe414bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6fe8c87700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe40b50d0 0x7f6fe40c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52470/0 (socket says 172.21.15.116:52470) 2023-11-08T08:18:49.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6fe8c87700 1 -- 172.21.15.116:0/342710235 learned_addr learned my addr 172.21.15.116:0/342710235 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f6fe3fff700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fe415a500 0x7f6fe406a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe8164ac0 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe8160e90 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe40117c0 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4011b70 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218011630 con 0x7fa22806cdd0 2023-11-08T08:18:49.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2280cb2d0 con 0x7fa22806cdd0 2023-11-08T08:18:49.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2280caff0 con 0x7fa22806cdd0 2023-11-08T08:18:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7f6fe3fff700 1 -- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe40b50d0 msgr2=0x7f6fe40c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7f6fe3fff700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe40b50d0 0x7f6fe40c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7f6fe3fff700 1 -- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe414b750 msgr2=0x7f6fe414bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7f6fe3fff700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe414b750 0x7f6fe414bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7f6fe3fff700 1 -- 172.21.15.116:0/342710235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80094a0 con 0x7f6fe415a500 2023-11-08T08:18:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22806b810 con 0x7fa22806cdd0 2023-11-08T08:18:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd00052f0 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f47d7fff700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 0x7f47d8206cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47c4005af0 tx=0x7f47c400ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4010b90 con 0x7f47d80ae020 2023-11-08T08:18:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c400dd00 con 0x7f47d80ae020 2023-11-08T08:18:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4019680 con 0x7f47d80ae020 2023-11-08T08:18:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d80b0de0 con 0x7f47d80ae020 2023-11-08T08:18:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.063+0000 7fe5f93f8700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5d80682f0 0x7fe5d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.062+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d80b0c50 con 0x7f47d80ae020 2023-11-08T08:18:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f04e3691700 1 -- 172.21.15.116:0/767036646 shutdown_connections 2023-11-08T08:18:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f6fe3fff700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fe415a500 0x7f6fe406a4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0000c00 tx=0x7f6fd000c780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b80052f0 con 0x7f47d80ae020 2023-11-08T08:18:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7fe5f93f8700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5d80682f0 0x7fe5d806a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5f001e040 tx=0x7fe5f0018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21801e070 con 0x7fa22806cdd0 2023-11-08T08:18:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218006c80 con 0x7fa22806cdd0 2023-11-08T08:18:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f04e3691700 1 -- 172.21.15.116:0/767036646 wait complete. 2023-11-08T08:18:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47c401f020 con 0x7f47d80ae020 2023-11-08T08:18:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f47d57fa700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47bc0683f0 0x7f47bc06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.064+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f47c4086700 con 0x7f47d80ae020 2023-11-08T08:18:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.065+0000 7f04e3691700 1 Processor -- start 2023-11-08T08:18:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.065+0000 7f04e3691700 1 -- start start 2023-11-08T08:18:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e3691700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04dc06d230 0x7f04dc1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e3691700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc0b0c50 0x7f04dc165e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e3691700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 0x7f04dc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e3691700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc167870 con 0x7f04dc158980 2023-11-08T08:18:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e3691700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc166540 con 0x7f04dc06d230 2023-11-08T08:18:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e3691700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc166820 con 0x7f04dc0b0c50 2023-11-08T08:18:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 0x7f04dc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 0x7f04dc0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39304/0 (socket says 172.21.15.116:39304) 2023-11-08T08:18:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 -- 172.21.15.116:0/1079316072 learned_addr learned my addr 172.21.15.116:0/1079316072 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e142d700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04dc06d230 0x7f04dc1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e0c2c700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc0b0c50 0x7f04dc165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 -- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc0b0c50 msgr2=0x7f04dc165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc0b0c50 0x7f04dc165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 -- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04dc06d230 msgr2=0x7f04dc1580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04dc06d230 0x7f04dc1580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 -- 172.21.15.116:0/1079316072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0094a0 con 0x7f04dc158980 2023-11-08T08:18:49.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e0c2c700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc0b0c50 0x7f04dc165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f04e1c2e700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 0x7f04dc0ceaa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04d400efc0 tx=0x7f04d400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f47dcc48700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47bc0683f0 0x7f47bc06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.066+0000 7f47dcc48700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47bc0683f0 0x7f47bc06a8a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0082f0 tx=0x7f47cc00d000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.067+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5ec0538a0 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.067+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa218017020 con 0x7fa22806cdd0 2023-11-08T08:18:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.067+0000 7fa224ff9700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2100682f0 0x7fa21006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.067+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa218087840 con 0x7fa22806cdd0 2023-11-08T08:18:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.067+0000 7fa2277fe700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2100682f0 0x7fa21006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.068+0000 7fa2277fe700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2100682f0 0x7fa21006a7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa2280b04a0 tx=0x7fa214006bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.069+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcbe400f050 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.069+0000 7fcbde7fc700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcbe4085a00 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d400cd40 con 0x7f04dc158980 2023-11-08T08:18:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d400f040 con 0x7f04dc158980 2023-11-08T08:18:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d40146c0 con 0x7f04dc158980 2023-11-08T08:18:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.071+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47c4052ea0 con 0x7f47d80ae020 2023-11-08T08:18:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc0cb090 con 0x7f04dc158980 2023-11-08T08:18:49.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.071+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc0cadb0 con 0x7f04dc158980 2023-11-08T08:18:49.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.072+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc158d50 con 0x7f04dc158980 2023-11-08T08:18:49.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.073+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04d4014820 con 0x7f04dc158980 2023-11-08T08:18:49.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.074+0000 7f04d1ffb700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04c00682f0 0x7f04c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.074+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f04d4086ff0 con 0x7f04dc158980 2023-11-08T08:18:49.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.070+0000 7fcbeddd2700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8014040 tx=0x7fcbd801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.075+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa218057a60 con 0x7fa22806cdd0 2023-11-08T08:18:49.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.077+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0004c80 con 0x7f6fe415a500 2023-11-08T08:18:49.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.077+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd0004de0 con 0x7f6fe415a500 2023-11-08T08:18:49.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.077+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00072f0 con 0x7f6fe415a500 2023-11-08T08:18:49.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.078+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40ce000 con 0x7f6fe415a500 2023-11-08T08:18:49.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.078+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe40cdd20 con 0x7f6fe415a500 2023-11-08T08:18:49.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.079+0000 7f04e142d700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04c00682f0 0x7f04c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.080+0000 7f04e142d700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04c00682f0 0x7f04c006a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005910 tx=0x7f04d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.080+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6fd001a020 con 0x7f6fe415a500 2023-11-08T08:18:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.080+0000 7f6fe17fa700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fc8068340 0x7f6fc806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.080+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6fd00857a0 con 0x7f6fe415a500 2023-11-08T08:18:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.080+0000 7f6fe8c87700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fc8068340 0x7f6fc806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.081+0000 7f6fe8c87700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fc8068340 0x7f6fc806a7f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009b90 tx=0x7f6fd8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.081+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe406baf0 con 0x7f6fe415a500 2023-11-08T08:18:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.082+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04d4053890 con 0x7f04dc158980 2023-11-08T08:18:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.087+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6fd0051f90 con 0x7f6fe415a500 2023-11-08T08:18:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.096+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcbe40521f0 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.440+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f81b40743f0 con 0x7f81b40c6800 2023-11-08T08:18:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.442+0000 7f819b7fe700 1 -- 172.21.15.116:0/3380927863 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81ac019020 con 0x7f81b40c6800 2023-11-08T08:18:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.441+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe5f4067d70 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.442+0000 7fe5e9ffb700 1 -- 172.21.15.116:0/2459824090 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe5ec056ec0 con 0x7fe5f40b5dc0 2023-11-08T08:18:49.444 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:49.448 INFO:teuthology.orchestra.run.smithi116.stdout:55834574851 2023-11-08T08:18:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5d80682f0 msgr2=0x7fe5d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5d80682f0 0x7fe5d806a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5f001e040 tx=0x7fe5f0018000 comp rx=0 tx=0).stop 2023-11-08T08:18:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 msgr2=0x7fe5f40c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 0x7fe5f40c6ea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009f60 tx=0x7fe5ec009f90 comp rx=0 tx=0).stop 2023-11-08T08:18:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f65140743f0 con 0x7f65140c25d0 2023-11-08T08:18:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 shutdown_connections 2023-11-08T08:18:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f415c8c0 0x7fe5f41545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5d80682f0 0x7fe5d806a7a0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5f001e040 tx=0x7fe5f0018000 comp rx=0 tx=0).stop 2023-11-08T08:18:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5f406b050 0x7fe5f40c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 --2- 172.21.15.116:0/2459824090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5f40b5dc0 0x7fe5f40c6ea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 >> 172.21.15.116:0/2459824090 conn(0x7fe5f406c520 msgr2=0x7fe5f40b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.449+0000 7f64fbfff700 1 -- 172.21.15.116:0/1507874002 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f650c021090 con 0x7f65140c25d0 2023-11-08T08:18:49.453 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:49.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81a0068340 msgr2=0x7f81a006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81a0068340 0x7f81a006a7f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f81b4070890 tx=0x7f81a8010000 comp rx=0 tx=0).stop 2023-11-08T08:18:49.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 msgr2=0x7f81b40bf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 0x7f81b40bf8f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00b990 tx=0x7f81ac00bd50 comp rx=0 tx=0).stop 2023-11-08T08:18:49.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 shutdown_connections 2023-11-08T08:18:49.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7fe5fb65c700 1 -- 172.21.15.116:0/2459824090 wait complete. 2023-11-08T08:18:49.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 shutdown_connections 2023-11-08T08:18:49.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81a0068340 0x7f81a006a7f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b40c6800 0x7f81b40bf8f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b40b2d10 0x7f81b4160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 --2- 172.21.15.116:0/3380927863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4161b40 0x7f81b40c1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.451+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 >> 172.21.15.116:0/3380927863 conn(0x7f81b406ea80 msgr2=0x7f81b4163360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.452+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 shutdown_connections 2023-11-08T08:18:49.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.452+0000 7f81b9195700 1 -- 172.21.15.116:0/3380927863 wait complete. 2023-11-08T08:18:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.452+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65000682f0 msgr2=0x7f650006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.452+0000 7f6519e9d700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65000682f0 0x7f650006a7a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f651415f360 tx=0x7f6508008040 comp rx=0 tx=0).stop 2023-11-08T08:18:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.452+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 msgr2=0x7f65140c2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.452+0000 7f6519e9d700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 0x7f65140c2a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00c2a0 tx=0x7f650c001650 comp rx=0 tx=0).stop 2023-11-08T08:18:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 shutdown_connections 2023-11-08T08:18:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f6519e9d700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651415f5d0 0x7f651415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f6519e9d700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65000682f0 0x7f650006a7a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f6519e9d700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64fc004850 0x7f651415e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f6519e9d700 1 --2- 172.21.15.116:0/1507874002 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65140c25d0 0x7f65140c2a40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 >> 172.21.15.116:0/1507874002 conn(0x7f651406aed0 msgr2=0x7f65141602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.453+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f47b80055a0 con 0x7f47d80ae020 2023-11-08T08:18:49.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.462+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa2280743f0 con 0x7fa22806cdd0 2023-11-08T08:18:49.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.462+0000 7f47d57fa700 1 -- 172.21.15.116:0/2734097085 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f47c40564c0 con 0x7f47d80ae020 2023-11-08T08:18:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.462+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 shutdown_connections 2023-11-08T08:18:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.462+0000 7fa224ff9700 1 -- 172.21.15.116:0/2048436995 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa218020020 con 0x7fa22806cdd0 2023-11-08T08:18:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.462+0000 7f6519e9d700 1 -- 172.21.15.116:0/1507874002 wait complete. 2023-11-08T08:18:49.464 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:49.464 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-08T08:18:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.466+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47bc0683f0 msgr2=0x7f47bc06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.466+0000 7f47deeac700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47bc0683f0 0x7f47bc06a8a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0082f0 tx=0x7f47cc00d000 comp rx=0 tx=0).stop 2023-11-08T08:18:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.466+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 msgr2=0x7f47d8206cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.466+0000 7f47deeac700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 0x7f47d8206cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47c4005af0 tx=0x7f47c400ae20 comp rx=0 tx=0).stop 2023-11-08T08:18:49.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2023-11-08T08:18:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2100682f0 msgr2=0x7fa21006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7fa22e382700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2100682f0 0x7fa21006a7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa2280b04a0 tx=0x7fa214006bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 msgr2=0x7fa2280c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7fa22e382700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 0x7fa2280c2fe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa22806f8e0 tx=0x7fa218004750 comp rx=0 tx=0).stop 2023-11-08T08:18:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 shutdown_connections 2023-11-08T08:18:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47d8207200 0x7f47d80cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47bc0683f0 0x7f47bc06a8a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d806c930 0x7f47d80c2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 --2- 172.21.15.116:0/2734097085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47d80ae020 0x7f47d8206cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 >> 172.21.15.116:0/2734097085 conn(0x7f47d806aed0 msgr2=0x7f47d816d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 shutdown_connections 2023-11-08T08:18:49.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.467+0000 7f47deeac700 1 -- 172.21.15.116:0/2734097085 wait complete. 2023-11-08T08:18:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.469+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f04dc0743f0 con 0x7f04dc158980 2023-11-08T08:18:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.470+0000 7f04d1ffb700 1 -- 172.21.15.116:0/1079316072 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f04d4009ca0 con 0x7f04dc158980 2023-11-08T08:18:49.472 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.471+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 shutdown_connections 2023-11-08T08:18:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.471+0000 7fa22e382700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2281609c0 0x7fa2281f9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.471+0000 7fa22e382700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2100682f0 0x7fa21006a7a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.471+0000 7fa22e382700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa22806cdd0 0x7fa2280c2fe0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.471+0000 7fa22e382700 1 --2- 172.21.15.116:0/2048436995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2280b0c50 0x7fa2280afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.471+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 >> 172.21.15.116:0/2048436995 conn(0x7fa22806aed0 msgr2=0x7fa2281645c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.473+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 shutdown_connections 2023-11-08T08:18:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.473+0000 7fa22e382700 1 -- 172.21.15.116:0/2048436995 wait complete. 2023-11-08T08:18:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.473+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04c00682f0 msgr2=0x7f04c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.473+0000 7f04e3691700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04c00682f0 0x7f04c006a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005910 tx=0x7f04d8009040 comp rx=0 tx=0).stop 2023-11-08T08:18:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.473+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 msgr2=0x7f04dc0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.473+0000 7f04e3691700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 0x7f04dc0ceaa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04d400efc0 tx=0x7f04d400c3a0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 shutdown_connections 2023-11-08T08:18:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04dc158980 0x7f04dc0ceaa0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f04c00682f0 0x7f04c006a7a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04dc06d230 0x7f04dc1580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 --2- 172.21.15.116:0/1079316072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f04dc0b0c50 0x7f04dc165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 >> 172.21.15.116:0/1079316072 conn(0x7f04dc06aed0 msgr2=0x7f04dc0ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 shutdown_connections 2023-11-08T08:18:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.474+0000 7f04e3691700 1 -- 172.21.15.116:0/1079316072 wait complete. 2023-11-08T08:18:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.476+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6fe40743f0 con 0x7f6fe415a500 2023-11-08T08:18:49.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2023-11-08T08:18:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.476+0000 7f6fe17fa700 1 -- 172.21.15.116:0/342710235 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6fd00555b0 con 0x7f6fe415a500 2023-11-08T08:18:49.480 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.478+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcbd0005870 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.479+0000 7fcbde7fc700 1 -- 172.21.15.116:0/2610960444 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcbe4055810 con 0x7fcbe80b7cd0 2023-11-08T08:18:49.481 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-08T08:18:49.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.480+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fc8068340 msgr2=0x7f6fc806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.480+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fc8068340 0x7f6fc806a7f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009b90 tx=0x7f6fd8008030 comp rx=0 tx=0).stop 2023-11-08T08:18:49.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.480+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fe415a500 msgr2=0x7f6fe406a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.480+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fe415a500 0x7f6fe406a4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0000c00 tx=0x7f6fd000c780 comp rx=0 tx=0).stop 2023-11-08T08:18:49.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 shutdown_connections 2023-11-08T08:18:49.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe414b750 0x7f6fe414bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6fe40b50d0 0x7f6fe40c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6fc8068340 0x7f6fc806a7f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 --2- 172.21.15.116:0/342710235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fe415a500 0x7f6fe406a4e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 >> 172.21.15.116:0/342710235 conn(0x7f6fe40cf400 msgr2=0x7f6fe406c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 shutdown_connections 2023-11-08T08:18:49.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7f6feaeeb700 1 -- 172.21.15.116:0/342710235 wait complete. 2023-11-08T08:18:49.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcbcc0682f0 msgr2=0x7fcbcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8014040 tx=0x7fcbd801c040 comp rx=0 tx=0).stop 2023-11-08T08:18:49.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 msgr2=0x7fcbe8151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:49.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.481+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 0x7fcbe8151610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b3c0 tx=0x7fcbe400b6d0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.482+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 shutdown_connections 2023-11-08T08:18:49.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.482+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe80c6aa0 0x7fcbe80c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.482+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.482+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd4003a00 0x7fcbe81f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.482+0000 7fcbf0036700 1 --2- 172.21.15.116:0/2610960444 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbe80b7cd0 0x7fcbe8151610 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:49.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.482+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 >> 172.21.15.116:0/2610960444 conn(0x7fcbe806c520 msgr2=0x7fcbe80b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:49.488 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2023-11-08T08:18:49.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.483+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 shutdown_connections 2023-11-08T08:18:49.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:49.483+0000 7fcbf0036700 1 -- 172.21.15.116:0/2610960444 wait complete. 2023-11-08T08:18:49.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2023-11-08T08:18:49.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2023-11-08T08:18:49.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2023-11-08T08:18:49.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-11-08T08:18:49.502 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2023-11-08T08:18:50.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-08T08:18:50.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-08T08:18:50.489 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-08T08:18:50.490 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-08T08:18:50.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-08T08:18:50.500 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-08T08:18:50.501 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-08T08:18:50.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-08T08:18:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.665+0000 7f0c17bca700 1 Processor -- start 2023-11-08T08:18:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.665+0000 7f0c17bca700 1 -- start start 2023-11-08T08:18:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.666+0000 7f0c17bca700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c100ae020 0x7f0c1006c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.666+0000 7f0c17bca700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0c1006c930 0x7f0c100b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.666+0000 7f0c17bca700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100cb160 con 0x7f0c1008ac20 2023-11-08T08:18:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.666+0000 7f0c17bca700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c1006b910 con 0x7f0c100ae020 2023-11-08T08:18:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.666+0000 7f0c17bca700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c1006bbf0 con 0x7f0c1006c930 2023-11-08T08:18:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.667+0000 7f0c15165700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f0c1008ac20 0x7f0c100cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57324/0 (socket says 172.21.15.116:57324) 2023-11-08T08:18:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.667+0000 7f0c15165700 1 -- 172.21.15.116:0/168535568 learned_addr learned my addr 172.21.15.116:0/168535568 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.667+0000 7f0c16167700 1 --2- 172.21.15.116:0/168535568 >> v2:172.21.15.129:3300/0 conn(0x7f0c1006c930 0x7f0c100b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.668+0000 7f0c15966700 1 --2- 172.21.15.116:0/168535568 >> v2:172.21.15.116:3300/0 conn(0x7f0c100ae020 0x7f0c1006c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.668+0000 7f0c15966700 1 -- 172.21.15.116:0/168535568 >> v2:172.21.15.129:3300/0 conn(0x7f0c1006c930 msgr2=0x7f0c100b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.668+0000 7f0c15966700 1 --2- 172.21.15.116:0/168535568 >> v2:172.21.15.129:3300/0 conn(0x7f0c1006c930 0x7f0c100b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.668+0000 7f0c15966700 1 -- 172.21.15.116:0/168535568 >> v1:172.21.15.129:6789/0 conn(0x7f0c1008ac20 legacy=0x7f0c100cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.668+0000 7f0c15966700 1 -- 172.21.15.116:0/168535568 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1006bed0 con 0x7f0c100ae020 2023-11-08T08:18:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.668+0000 7f0c15966700 1 --2- 172.21.15.116:0/168535568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c100ae020 0x7f0c1006c3f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0c1006f8b0 tx=0x7f0c00009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5710739497923c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.669+0000 7f0c14964700 1 -- 172.21.15.116:0/168535568 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00004230 con 0x7f0c100ae020 2023-11-08T08:18:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.669+0000 7f0c14964700 1 -- 172.21.15.116:0/168535568 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00004390 con 0x7f0c100ae020 2023-11-08T08:18:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.669+0000 7f0c14964700 1 -- 172.21.15.116:0/168535568 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c000036c0 con 0x7f0c100ae020 2023-11-08T08:18:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.670+0000 7f0c17bca700 1 -- 172.21.15.116:0/168535568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c100ae020 msgr2=0x7f0c1006c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.670+0000 7f0c17bca700 1 --2- 172.21.15.116:0/168535568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c100ae020 0x7f0c1006c3f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0c1006f8b0 tx=0x7f0c00009b20 comp rx=0 tx=0).stop 2023-11-08T08:18:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.671+0000 7f0c17bca700 1 -- 172.21.15.116:0/168535568 shutdown_connections 2023-11-08T08:18:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.671+0000 7f0c17bca700 1 --2- 172.21.15.116:0/168535568 >> v2:172.21.15.129:3300/0 conn(0x7f0c1006c930 0x7f0c100b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.671+0000 7f0c17bca700 1 --2- 172.21.15.116:0/168535568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c100ae020 0x7f0c1006c3f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.671+0000 7f0c17bca700 1 -- 172.21.15.116:0/168535568 >> 172.21.15.116:0/168535568 conn(0x7f0c1006aed0 msgr2=0x7f0c100cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.671+0000 7f0c17bca700 1 -- 172.21.15.116:0/168535568 shutdown_connections 2023-11-08T08:18:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.671+0000 7f0c17bca700 1 -- 172.21.15.116:0/168535568 wait complete. 2023-11-08T08:18:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.673+0000 7f0c17bca700 1 Processor -- start 2023-11-08T08:18:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.673+0000 7f0c17bca700 1 -- start start 2023-11-08T08:18:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c17bca700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 0x7f0c100b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c17bca700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0c100ae020 0x7f0c100b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c17bca700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c100810f0 0x7f0c10081840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c17bca700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c100c2170 con 0x7f0c1006c930 2023-11-08T08:18:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c17bca700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c10169840 con 0x7f0c100810f0 2023-11-08T08:18:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c17bca700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c100826a0 con 0x7f0c100ae020 2023-11-08T08:18:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c15966700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 0x7f0c100b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c15966700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 0x7f0c100b18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39322/0 (socket says 172.21.15.116:39322) 2023-11-08T08:18:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c15966700 1 -- 172.21.15.116:0/1345035584 learned_addr learned my addr 172.21.15.116:0/1345035584 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.675+0000 7f0c15165700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0c100ae020 0x7f0c100b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c16167700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c100810f0 0x7f0c10081840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c15966700 1 -- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0c100ae020 msgr2=0x7f0c100b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c15966700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0c100ae020 0x7f0c100b1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c15966700 1 -- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c100810f0 msgr2=0x7f0c10081840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c15966700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c100810f0 0x7f0c10081840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c15966700 1 -- 172.21.15.116:0/1345035584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000094a0 con 0x7f0c1006c930 2023-11-08T08:18:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.676+0000 7f0c15966700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 0x7f0c100b18f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0c10067450 tx=0x7f0c00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.678+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00011630 con 0x7f0c1006c930 2023-11-08T08:18:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.678+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00011790 con 0x7f0c1006c930 2023-11-08T08:18:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.678+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00011b00 con 0x7f0c1006c930 2023-11-08T08:18:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.678+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c100ccf30 con 0x7f0c1006c930 2023-11-08T08:18:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.678+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c100ccda0 con 0x7f0c1006c930 2023-11-08T08:18:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.681+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf80052f0 con 0x7f0c1006c930 2023-11-08T08:18:50.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.692+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c0002e070 con 0x7f0c1006c930 2023-11-08T08:18:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.692+0000 7f0c067fc700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bfc068520 0x7f0bfc06a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.692+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0c00087470 con 0x7f0c1006c930 2023-11-08T08:18:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.693+0000 7f0c15165700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bfc068520 0x7f0bfc06a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.693+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c00053c60 con 0x7f0c1006c930 2023-11-08T08:18:50.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.693+0000 7f0c15165700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bfc068520 0x7f0bfc06a9d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0c101675f0 tx=0x7f0c08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.726+0000 7fa7c40d8700 1 Processor -- start 2023-11-08T08:18:50.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.726+0000 7fa7c40d8700 1 -- start start 2023-11-08T08:18:50.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c40d8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7bc0cf710 0x7fa7bc0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c40d8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa7bc06a980 0x7fa7bc071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c40d8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc15cc10 con 0x7fa7bc071890 2023-11-08T08:18:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c40d8700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc06c9b0 con 0x7fa7bc0cf710 2023-11-08T08:18:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c40d8700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc06cc90 con 0x7fa7bc06a980 2023-11-08T08:18:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c1673700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa7bc071890 0x7fa7bc06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57334/0 (socket says 172.21.15.116:57334) 2023-11-08T08:18:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.727+0000 7fa7c1673700 1 -- 172.21.15.116:0/4104895770 learned_addr learned my addr 172.21.15.116:0/4104895770 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:50.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.728+0000 7fa7c2675700 1 --2- 172.21.15.116:0/4104895770 >> v2:172.21.15.129:3300/0 conn(0x7fa7bc06a980 0x7fa7bc071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.729+0000 7fa7c2675700 1 -- 172.21.15.116:0/4104895770 >> v2:172.21.15.116:3300/0 conn(0x7fa7bc0cf710 msgr2=0x7fa7bc0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.729+0000 7fa7c2675700 1 --2- 172.21.15.116:0/4104895770 >> v2:172.21.15.116:3300/0 conn(0x7fa7bc0cf710 0x7fa7bc0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.729+0000 7fa7c2675700 1 -- 172.21.15.116:0/4104895770 >> v1:172.21.15.129:6789/0 conn(0x7fa7bc071890 legacy=0x7fa7bc06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.729+0000 7fa7c2675700 1 -- 172.21.15.116:0/4104895770 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc06cf70 con 0x7fa7bc06a980 2023-11-08T08:18:50.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.730+0000 7f17bdbf9700 1 Processor -- start 2023-11-08T08:18:50.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.730+0000 7fa7c2675700 1 --2- 172.21.15.116:0/4104895770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc06a980 0x7fa7bc071220 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009d90 tx=0x7fa7b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3534b8795ba70bb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.730+0000 7f17bdbf9700 1 -- start start 2023-11-08T08:18:50.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.730+0000 7fa7c0e72700 1 -- 172.21.15.116:0/4104895770 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400e070 con 0x7fa7bc06a980 2023-11-08T08:18:50.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.730+0000 7fa7c0e72700 1 -- 172.21.15.116:0/4104895770 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b400bad0 con 0x7fa7bc06a980 2023-11-08T08:18:50.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.730+0000 7fa7c0e72700 1 -- 172.21.15.116:0/4104895770 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400be60 con 0x7fa7bc06a980 2023-11-08T08:18:50.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7fa7c40d8700 1 -- 172.21.15.116:0/4104895770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc06a980 msgr2=0x7fa7bc071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/4104895770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc06a980 0x7fa7bc071220 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009d90 tx=0x7fa7b400b330 comp rx=0 tx=0).stop 2023-11-08T08:18:50.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17bdbf9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17b8160c30 0x7f17b80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17bdbf9700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b80c2460 con 0x7f17b80b0ef0 2023-11-08T08:18:50.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17bdbf9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b806b770 con 0x7f17b80adda0 2023-11-08T08:18:50.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17bdbf9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b80ae240 con 0x7f17b8160c30 2023-11-08T08:18:50.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7fa7c40d8700 1 -- 172.21.15.116:0/4104895770 shutdown_connections 2023-11-08T08:18:50.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/4104895770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc06a980 0x7fa7bc071220 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/4104895770 >> v2:172.21.15.116:3300/0 conn(0x7fa7bc0cf710 0x7fa7bc0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7fa7c40d8700 1 -- 172.21.15.116:0/4104895770 >> 172.21.15.116:0/4104895770 conn(0x7fa7bc06c010 msgr2=0x7fa7bc0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17b7fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17b80b0ef0 0x7f17b80ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34348/0 (socket says 172.21.15.116:34348) 2023-11-08T08:18:50.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17b7fff700 1 -- 172.21.15.116:0/4072299386 learned_addr learned my addr 172.21.15.116:0/4072299386 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:50.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.732+0000 7fa7c40d8700 1 -- 172.21.15.116:0/4104895770 shutdown_connections 2023-11-08T08:18:50.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.731+0000 7f17b77fe700 1 --2- 172.21.15.116:0/4072299386 >> v2:172.21.15.116:3300/0 conn(0x7f17b8160c30 0x7f17b80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.732+0000 7f17b77fe700 1 -- 172.21.15.116:0/4072299386 >> v1:172.21.15.129:6789/0 conn(0x7f17b80adda0 legacy=0x7f17b80b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.732+0000 7f17b77fe700 1 -- 172.21.15.116:0/4072299386 >> v1:172.21.15.116:6789/0 conn(0x7f17b80b0ef0 legacy=0x7f17b80ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.732+0000 7f17b77fe700 1 -- 172.21.15.116:0/4072299386 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8161000 con 0x7f17b8160c30 2023-11-08T08:18:50.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.732+0000 7fa7c40d8700 1 -- 172.21.15.116:0/4104895770 wait complete. 2023-11-08T08:18:50.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.732+0000 7f17b77fe700 1 --2- 172.21.15.116:0/4072299386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b8160c30 0x7f17b80ad860 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009840 tx=0x7f17ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a11861080ac9541 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.733+0000 7f17b67fc700 1 -- 172.21.15.116:0/4072299386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac004230 con 0x7f17b8160c30 2023-11-08T08:18:50.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.733+0000 7f17b67fc700 1 -- 172.21.15.116:0/4072299386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ac004390 con 0x7f17b8160c30 2023-11-08T08:18:50.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.733+0000 7f17b67fc700 1 -- 172.21.15.116:0/4072299386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac0036a0 con 0x7f17b8160c30 2023-11-08T08:18:50.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.733+0000 7f36129d1700 1 Processor -- start 2023-11-08T08:18:50.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.733+0000 7fa7c40d8700 1 Processor -- start 2023-11-08T08:18:50.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 -- 172.21.15.116:0/4072299386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b8160c30 msgr2=0x7f17b80ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/4072299386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b8160c30 0x7f17b80ad860 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009840 tx=0x7f17ac009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:50.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 -- 172.21.15.116:0/4072299386 shutdown_connections 2023-11-08T08:18:50.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/4072299386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b8160c30 0x7f17b80ad860 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009840 tx=0x7f17ac009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:50.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 -- 172.21.15.116:0/4072299386 >> 172.21.15.116:0/4072299386 conn(0x7f17b80cb540 msgr2=0x7f17b80cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 -- 172.21.15.116:0/4072299386 shutdown_connections 2023-11-08T08:18:50.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7fa7c40d8700 1 -- start start 2023-11-08T08:18:50.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c40d8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc06a980 0x7fa7bc15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c40d8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7bc071890 0x7fa7bc0b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c40d8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc0cf710 0x7fa7bc15e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c40d8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b4670 con 0x7fa7bc06a980 2023-11-08T08:18:50.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c40d8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b54b0 con 0x7fa7bc0cf710 2023-11-08T08:18:50.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c40d8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b5940 con 0x7fa7bc071890 2023-11-08T08:18:50.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c1e74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc06a980 0x7fa7bc15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c1e74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc06a980 0x7fa7bc15b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39342/0 (socket says 172.21.15.116:39342) 2023-11-08T08:18:50.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c1e74700 1 -- 172.21.15.116:0/3818642238 learned_addr learned my addr 172.21.15.116:0/3818642238 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c2675700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc0cf710 0x7fa7bc15e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7fa7c1673700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7bc071890 0x7fa7bc0b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c2675700 1 -- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7bc071890 msgr2=0x7fa7bc0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c2675700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7bc071890 0x7fa7bc0b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c2675700 1 -- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc06a980 msgr2=0x7fa7bc15b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c2675700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc06a980 0x7fa7bc15b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c2675700 1 -- 172.21.15.116:0/3818642238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b4009a70 con 0x7fa7bc0cf710 2023-11-08T08:18:50.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c2675700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc0cf710 0x7fa7bc15e3f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc06b0f0 tx=0x7fa7b4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.733+0000 7f36129d1700 1 -- start start 2023-11-08T08:18:50.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f36129d1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f360c06b220 0x7f360c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f36129d1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360c0d22c0 con 0x7f360c160ac0 2023-11-08T08:18:50.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f36129d1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360c0cf550 con 0x7f360c1652c0 2023-11-08T08:18:50.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f36129d1700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360c0d0840 con 0x7f360c06b220 2023-11-08T08:18:50.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f360b7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f360c06b220 0x7f360c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f360b7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f360c06b220 0x7f360c0cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39358/0 (socket says 172.21.15.116:39358) 2023-11-08T08:18:50.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7f360b7fe700 1 -- 172.21.15.116:0/626760985 learned_addr learned my addr 172.21.15.116:0/626760985 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7efd0958a700 1 Processor -- start 2023-11-08T08:18:50.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.735+0000 7efd0958a700 1 -- start start 2023-11-08T08:18:50.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.734+0000 7f17bdbf9700 1 -- 172.21.15.116:0/4072299386 wait complete. 2023-11-08T08:18:50.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7f17bdbf9700 1 Processor -- start 2023-11-08T08:18:50.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7f17bdbf9700 1 -- start start 2023-11-08T08:18:50.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7f360b7fe700 1 -- 172.21.15.116:0/626760985 >> v1:172.21.15.129:6789/0 conn(0x7f360c1652c0 legacy=0x7f360c0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7f360b7fe700 1 -- 172.21.15.116:0/626760985 >> v1:172.21.15.116:6789/0 conn(0x7f360c160ac0 legacy=0x7f360c06ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7f360b7fe700 1 -- 172.21.15.116:0/626760985 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c1647e0 con 0x7f360c06b220 2023-11-08T08:18:50.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f360b7fe700 1 --2- 172.21.15.116:0/626760985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c06b220 0x7f360c0cef20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36000098f0 tx=0x7f3600009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3f30149de0f33 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4012650 con 0x7fa7bc0cf710 2023-11-08T08:18:50.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b40127b0 con 0x7fa7bc0cf710 2023-11-08T08:18:50.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4012b60 con 0x7fa7bc0cf710 2023-11-08T08:18:50.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc06d530 con 0x7fa7bc0cf710 2023-11-08T08:18:50.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc06d250 con 0x7fa7bc0cf710 2023-11-08T08:18:50.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7efd0958a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd0408ac20 0x7efd040addb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7efd0958a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efd0408a5d0 0x7efd040cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7efd0958a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd040ae2f0 con 0x7efd040cac10 2023-11-08T08:18:50.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7efd0958a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0406b910 con 0x7efd0408ac20 2023-11-08T08:18:50.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.736+0000 7efd0958a700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0406bbf0 con 0x7efd0408a5d0 2023-11-08T08:18:50.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7efd02ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd0408ac20 0x7efd040addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7efd02ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd0408ac20 0x7efd040addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39368/0 (socket says 172.21.15.116:39368) 2023-11-08T08:18:50.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7efd02ffd700 1 -- 172.21.15.116:0/1170760974 learned_addr learned my addr 172.21.15.116:0/1170760974 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7efd037fe700 1 --2- 172.21.15.116:0/1170760974 >> v2:172.21.15.129:3300/0 conn(0x7efd0408a5d0 0x7efd040cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f17bdbf9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b80adda0 0x7f17b8202200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f17bdbf9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 0x7f17b80af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f17bdbf9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b80afab0 0x7f17b8202de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f17bdbf9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b80aff30 con 0x7f17b80afab0 2023-11-08T08:18:50.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f17bdbf9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b81611f0 con 0x7f17b80adda0 2023-11-08T08:18:50.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.737+0000 7f17bdbf9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b8162200 con 0x7f17b80b0ef0 2023-11-08T08:18:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7efd02ffd700 1 -- 172.21.15.116:0/1170760974 >> v2:172.21.15.129:3300/0 conn(0x7efd0408a5d0 msgr2=0x7efd040cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7efd02ffd700 1 --2- 172.21.15.116:0/1170760974 >> v2:172.21.15.129:3300/0 conn(0x7efd0408a5d0 0x7efd040cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7efd02ffd700 1 -- 172.21.15.116:0/1170760974 >> v1:172.21.15.129:6789/0 conn(0x7efd040cac10 legacy=0x7efd0406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7efd02ffd700 1 -- 172.21.15.116:0/1170760974 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0406bed0 con 0x7efd0408ac20 2023-11-08T08:18:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f360affd700 1 -- 172.21.15.116:0/626760985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600004230 con 0x7f360c06b220 2023-11-08T08:18:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 0x7f17b80af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 0x7f17b80af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52512/0 (socket says 172.21.15.116:52512) 2023-11-08T08:18:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 -- 172.21.15.116:0/642282618 learned_addr learned my addr 172.21.15.116:0/642282618 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f360affd700 1 -- 172.21.15.116:0/626760985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600004390 con 0x7f360c06b220 2023-11-08T08:18:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f360affd700 1 -- 172.21.15.116:0/626760985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36000036a0 con 0x7f360c06b220 2023-11-08T08:18:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b7fff700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b80afab0 0x7f17b8202de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7efd02ffd700 1 --2- 172.21.15.116:0/1170760974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0408ac20 0x7efd040addb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efd0406d490 tx=0x7efcf40147e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e407259d925cec2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 -- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b80adda0 msgr2=0x7f17b8202200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-08T08:18:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b80adda0 0x7f17b8202200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 -- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b80afab0 msgr2=0x7f17b8202de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b80afab0 0x7f17b8202de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.738+0000 7f17b6ffd700 1 -- 172.21.15.116:0/642282618 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac0094a0 con 0x7f17b80b0ef0 2023-11-08T08:18:50.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f36129d1700 1 -- 172.21.15.116:0/626760985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c06b220 msgr2=0x7f360c0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f36129d1700 1 --2- 172.21.15.116:0/626760985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c06b220 0x7f360c0cef20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36000098f0 tx=0x7f3600009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:50.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f17b6ffd700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 0x7f17b80af570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17a800c4d0 tx=0x7f17a800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7efd027fc700 1 -- 172.21.15.116:0/1170760974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4005e30 con 0x7efd0408ac20 2023-11-08T08:18:50.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7efd027fc700 1 -- 172.21.15.116:0/1170760974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf400f0d0 con 0x7efd0408ac20 2023-11-08T08:18:50.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7efd027fc700 1 -- 172.21.15.116:0/1170760974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4015640 con 0x7efd0408ac20 2023-11-08T08:18:50.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f36129d1700 1 -- 172.21.15.116:0/626760985 shutdown_connections 2023-11-08T08:18:50.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f36129d1700 1 --2- 172.21.15.116:0/626760985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c06b220 0x7f360c0cef20 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36000098f0 tx=0x7f3600009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:50.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f36129d1700 1 -- 172.21.15.116:0/626760985 >> 172.21.15.116:0/626760985 conn(0x7f360c06c520 msgr2=0x7f360c06c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a80052d0 con 0x7f17b80b0ef0 2023-11-08T08:18:50.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a8005430 con 0x7f17b80b0ef0 2023-11-08T08:18:50.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a80038c0 con 0x7f17b80b0ef0 2023-11-08T08:18:50.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f36129d1700 1 -- 172.21.15.116:0/626760985 shutdown_connections 2023-11-08T08:18:50.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b8164a90 con 0x7f17b80b0ef0 2023-11-08T08:18:50.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b81647b0 con 0x7f17b80b0ef0 2023-11-08T08:18:50.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 -- 172.21.15.116:0/1170760974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0408ac20 msgr2=0x7efd040addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 --2- 172.21.15.116:0/1170760974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0408ac20 0x7efd040addb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efd0406d490 tx=0x7efcf40147e0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 -- 172.21.15.116:0/1170760974 shutdown_connections 2023-11-08T08:18:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 --2- 172.21.15.116:0/1170760974 >> v2:172.21.15.129:3300/0 conn(0x7efd0408a5d0 0x7efd040cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 --2- 172.21.15.116:0/1170760974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0408ac20 0x7efd040addb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 -- 172.21.15.116:0/1170760974 >> 172.21.15.116:0/1170760974 conn(0x7efd0406aed0 msgr2=0x7efd040cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7f36129d1700 1 -- 172.21.15.116:0/626760985 wait complete. 2023-11-08T08:18:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.739+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7b4023030 con 0x7fa7bc0cf710 2023-11-08T08:18:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7fa7b27fc700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7a00683c0 0x7fa7a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa7b4087590 con 0x7fa7bc0cf710 2023-11-08T08:18:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.741+0000 7fbd28853700 1 Processor -- start 2023-11-08T08:18:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.741+0000 7f36129d1700 1 Processor -- start 2023-11-08T08:18:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.741+0000 7fbd28853700 1 -- start start 2023-11-08T08:18:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.741+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b8164d70 con 0x7f17b80b0ef0 2023-11-08T08:18:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7fbd28853700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbd2415c810 0x7fbd24160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7fbd28853700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbd240718e0 0x7fbd2406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7fbd28853700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd241607a0 con 0x7fbd2406cdc0 2023-11-08T08:18:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7fbd28853700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2415cbe0 con 0x7fbd2415c810 2023-11-08T08:18:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7fbd28853700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd24160a50 con 0x7fbd240718e0 2023-11-08T08:18:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7f36129d1700 1 -- start start 2023-11-08T08:18:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f17a8019030 con 0x7f17b80b0ef0 2023-11-08T08:18:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7f179ffff700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17a40683c0 0x7f17a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f17a8084fa0 con 0x7f17b80b0ef0 2023-11-08T08:18:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 -- 172.21.15.116:0/1170760974 shutdown_connections 2023-11-08T08:18:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.740+0000 7efd0958a700 1 -- 172.21.15.116:0/1170760974 wait complete. 2023-11-08T08:18:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7efd0958a700 1 Processor -- start 2023-11-08T08:18:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.742+0000 7efd0958a700 1 -- start start 2023-11-08T08:18:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbd240718e0 0x7fbd2406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbd240718e0 0x7fbd2406c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:57322/0 (socket says 172.21.15.116:57322) 2023-11-08T08:18:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 -- 172.21.15.116:0/339182551 learned_addr learned my addr 172.21.15.116:0/339182551 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f36129d1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f360c160ac0 0x7f360c1f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f36129d1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 0x7f360c161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f36129d1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f360c15be20 0x7f360c0b7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f36129d1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c161890 con 0x7f360c1652c0 2023-11-08T08:18:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f36129d1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c0b74a0 con 0x7f360c15be20 2023-11-08T08:18:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f36129d1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360c0b6e90 con 0x7f360c160ac0 2023-11-08T08:18:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f360b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 0x7f360c161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 -- 172.21.15.116:0/339182551 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415c810 msgr2=0x7fbd24160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 --2- 172.21.15.116:0/339182551 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415c810 0x7fbd24160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 -- 172.21.15.116:0/339182551 >> v1:172.21.15.116:6789/0 conn(0x7fbd2406cdc0 legacy=0x7fbd2406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 -- 172.21.15.116:0/339182551 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24160d30 con 0x7fbd240718e0 2023-11-08T08:18:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f360b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 0x7f360c161350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39380/0 (socket says 172.21.15.116:39380) 2023-11-08T08:18:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f360b7fe700 1 -- 172.21.15.116:0/3122759409 learned_addr learned my addr 172.21.15.116:0/3122759409 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a40052f0 con 0x7fa7bc0cf710 2023-11-08T08:18:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd21d9b700 1 --2- 172.21.15.116:0/339182551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd240718e0 0x7fbd2406c880 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbd240cc380 tx=0x7fbd18009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caaba0046881ef2e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fc0af7d6700 1 Processor -- start 2023-11-08T08:18:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fa7c1e74700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7a00683c0 0x7fa7a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7f360bfff700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f360c160ac0 0x7f360c1f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7f3610f6e700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f360c15be20 0x7f360c0b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fbd2159a700 1 -- 172.21.15.116:0/339182551 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18004230 con 0x7fbd240718e0 2023-11-08T08:18:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7efd0958a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 0x7efd04089580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7efd0958a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cac10 0x7efd040c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7efd0958a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd040c3c10 0x7efd040c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7efd0958a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd040c5400 con 0x7efd040c3c10 2023-11-08T08:18:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7efd0958a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd040c5120 con 0x7efd040cac10 2023-11-08T08:18:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7efd0958a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd040c3130 con 0x7efd0408a5d0 2023-11-08T08:18:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.743+0000 7fc0af7d6700 1 -- start start 2023-11-08T08:18:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7fbd2159a700 1 -- 172.21.15.116:0/339182551 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd18004390 con 0x7fbd240718e0 2023-11-08T08:18:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7fbd2159a700 1 -- 172.21.15.116:0/339182551 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd180036a0 con 0x7fbd240718e0 2023-11-08T08:18:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7fa7c1e74700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7a00683c0 0x7fa7a006a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009ac0 tx=0x7fa7ac0091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7efd02ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 0x7efd04089580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7f17b77fe700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17a40683c0 0x7f17a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7efd02ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 0x7efd04089580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52524/0 (socket says 172.21.15.116:52524) 2023-11-08T08:18:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.744+0000 7efd02ffd700 1 -- 172.21.15.116:0/3663231206 learned_addr learned my addr 172.21.15.116:0/3663231206 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 -- 172.21.15.116:0/339182551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd240718e0 msgr2=0x7fbd2406c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 --2- 172.21.15.116:0/339182551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd240718e0 0x7fbd2406c880 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbd240cc380 tx=0x7fbd18009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 -- 172.21.15.116:0/339182551 shutdown_connections 2023-11-08T08:18:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 --2- 172.21.15.116:0/339182551 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415c810 0x7fbd24160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 --2- 172.21.15.116:0/339182551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd240718e0 0x7fbd2406c880 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 -- 172.21.15.116:0/339182551 >> 172.21.15.116:0/339182551 conn(0x7fbd2406c010 msgr2=0x7fbd240cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f360b7fe700 1 -- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f360c160ac0 msgr2=0x7f360c1f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f360b7fe700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f360c160ac0 0x7f360c1f6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f360b7fe700 1 -- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f360c15be20 msgr2=0x7f360c0b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f360b7fe700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f360c15be20 0x7f360c0b7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f360b7fe700 1 -- 172.21.15.116:0/3122759409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3600009550 con 0x7f360c1652c0 2023-11-08T08:18:50.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f17b77fe700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17a40683c0 0x7f17a406a870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f17b80b1630 tx=0x7f17ac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 -- 172.21.15.116:0/339182551 shutdown_connections 2023-11-08T08:18:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fbd28853700 1 -- 172.21.15.116:0/339182551 wait complete. 2023-11-08T08:18:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7efd02ffd700 1 -- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cac10 msgr2=0x7efd040c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7efd02ffd700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cac10 0x7efd040c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7efd02ffd700 1 -- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd040c3c10 msgr2=0x7efd040c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7efd02ffd700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd040c3c10 0x7efd040c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7efd02ffd700 1 -- 172.21.15.116:0/3663231206 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf400fd90 con 0x7efd0408a5d0 2023-11-08T08:18:50.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fc0af7d6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc0a80b0c50 0x7fc0a80adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fc0af7d6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a8158e20 con 0x7fc0a806d230 2023-11-08T08:18:50.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fc0af7d6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a80ca7f0 con 0x7fc0a80cbc50 2023-11-08T08:18:50.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fc0af7d6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a806c700 con 0x7fc0a80b0c50 2023-11-08T08:18:50.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fc0acd71700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc0a80cbc50 0x7fc0a80b0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57372/0 (socket says 172.21.15.116:57372) 2023-11-08T08:18:50.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7fc0acd71700 1 -- 172.21.15.116:0/2453117781 learned_addr learned my addr 172.21.15.116:0/2453117781 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:50.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7f360b7fe700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 0x7f360c161350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3600009c40 tx=0x7f3600004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.745+0000 7efd02ffd700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 0x7efd04089580 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efcf4000c00 tx=0x7efcf4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc0add73700 1 --2- 172.21.15.116:0/2453117781 >> v2:172.21.15.116:3300/0 conn(0x7fc0a80b0c50 0x7fc0a80adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600003b90 con 0x7f360c1652c0 2023-11-08T08:18:50.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc09ffff700 1 -- 172.21.15.116:0/2453117781 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140443278 0 0) 0x7fc0a8158e20 con 0x7fc0a806d230 2023-11-08T08:18:50.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc09ffff700 1 -- 172.21.15.116:0/2453117781 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0940035b0 con 0x7fc0a806d230 2023-11-08T08:18:50.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600020070 con 0x7f360c1652c0 2023-11-08T08:18:50.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c164da0 con 0x7f360c1652c0 2023-11-08T08:18:50.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36000045e0 con 0x7f360c1652c0 2023-11-08T08:18:50.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c164ac0 con 0x7f360c1652c0 2023-11-08T08:18:50.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc0add73700 1 -- 172.21.15.116:0/2453117781 >> v1:172.21.15.129:6789/0 conn(0x7fc0a80cbc50 legacy=0x7fc0a80b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc0add73700 1 -- 172.21.15.116:0/2453117781 >> v1:172.21.15.116:6789/0 conn(0x7fc0a806d230 legacy=0x7fc0a8158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc0add73700 1 -- 172.21.15.116:0/2453117781 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80caad0 con 0x7fc0a80b0c50 2023-11-08T08:18:50.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fc0add73700 1 --2- 172.21.15.116:0/2453117781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80adf60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009d90 tx=0x7fc0a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e4d87450eae6ae server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.746+0000 7fbd28853700 1 Processor -- start 2023-11-08T08:18:50.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.747+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec0052f0 con 0x7f360c1652c0 2023-11-08T08:18:50.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.747+0000 7fbd28853700 1 -- start start 2023-11-08T08:18:50.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.747+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17a8051790 con 0x7f17b80b0ef0 2023-11-08T08:18:50.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.747+0000 7fc09ffff700 1 -- 172.21.15.116:0/2453117781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a000e070 con 0x7fc0a80b0c50 2023-11-08T08:18:50.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.747+0000 7fc09ffff700 1 -- 172.21.15.116:0/2453117781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a000bae0 con 0x7fc0a80b0c50 2023-11-08T08:18:50.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.747+0000 7fc09ffff700 1 -- 172.21.15.116:0/2453117781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a000be90 con 0x7fc0a80b0c50 2023-11-08T08:18:50.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd28853700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbd2406cdc0 0x7fbd241554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7b4057860 con 0x7fa7bc0cf710 2023-11-08T08:18:50.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f360002b020 con 0x7f360c1652c0 2023-11-08T08:18:50.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7f3608ff9700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35f80682f0 0x7f35f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3600087510 con 0x7f360c1652c0 2023-11-08T08:18:50.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7f360bfff700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35f80682f0 0x7f35f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.750+0000 7f360bfff700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35f80682f0 0x7f35f806a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35fc007fd0 tx=0x7f35fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4019070 con 0x7efd0408a5d0 2023-11-08T08:18:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.751+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf401a040 con 0x7efd0408a5d0 2023-11-08T08:18:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.751+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4003cd0 con 0x7efd0408a5d0 2023-11-08T08:18:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.751+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd040ccbd0 con 0x7efd0408a5d0 2023-11-08T08:18:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.751+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0406cb00 con 0x7efd0408a5d0 2023-11-08T08:18:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.752+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd040ccd60 con 0x7efd0408a5d0 2023-11-08T08:18:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2453117781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80b0c50 msgr2=0x7fc0a80adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2453117781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80adf60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009d90 tx=0x7fc0a000b330 comp rx=0 tx=0).stop 2023-11-08T08:18:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.752+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2453117781 shutdown_connections 2023-11-08T08:18:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.752+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2453117781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80adf60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.752+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2453117781 >> 172.21.15.116:0/2453117781 conn(0x7fc0a806aed0 msgr2=0x7fc0a80cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd28853700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd240718e0 0x7fbd240b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd28853700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd2415c810 0x7fbd240aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd28853700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240b3410 con 0x7fbd240718e0 2023-11-08T08:18:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd28853700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240b3660 con 0x7fbd2415c810 2023-11-08T08:18:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd28853700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240b3940 con 0x7fbd2406cdc0 2023-11-08T08:18:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd2259c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbd2406cdc0 0x7fbd241554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd2259c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbd2406cdc0 0x7fbd241554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52532/0 (socket says 172.21.15.116:52532) 2023-11-08T08:18:50.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd2259c700 1 -- 172.21.15.116:0/901148030 learned_addr learned my addr 172.21.15.116:0/901148030 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd21d9b700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd240718e0 0x7fbd240b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.748+0000 7fbd22d9d700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd2415c810 0x7fbd240aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fbd21d9b700 1 -- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbd2406cdc0 msgr2=0x7fbd241554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fbd21d9b700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbd2406cdc0 0x7fbd241554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fbd21d9b700 1 -- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd2415c810 msgr2=0x7fbd240aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fbd21d9b700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd2415c810 0x7fbd240aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fbd21d9b700 1 -- 172.21.15.116:0/901148030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18009550 con 0x7fbd240718e0 2023-11-08T08:18:50.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.749+0000 7fbd21d9b700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd240718e0 0x7fbd240b50f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbd18015000 tx=0x7fbd18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.753+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2453117781 shutdown_connections 2023-11-08T08:18:50.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.753+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2453117781 wait complete. 2023-11-08T08:18:50.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.755+0000 7fc0af7d6700 1 Processor -- start 2023-11-08T08:18:50.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.757+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3600053d70 con 0x7f360c1652c0 2023-11-08T08:18:50.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.755+0000 7fc0af7d6700 1 -- start start 2023-11-08T08:18:50.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.756+0000 7fc0af7d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 0x7fc0a80c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.756+0000 7fc0af7d6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80703e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.756+0000 7fc0af7d6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a80cbc50 0x7fc0a80c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.756+0000 7fc0af7d6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c8cf0 con 0x7fc0a806d230 2023-11-08T08:18:50.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.756+0000 7fc0af7d6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c8070 con 0x7fc0a80b0c50 2023-11-08T08:18:50.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.756+0000 7fc0af7d6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c8350 con 0x7fc0a80cbc50 2023-11-08T08:18:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.757+0000 7fc0ad572700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 0x7fc0a80c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.757+0000 7fc0ad572700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 0x7fc0a80c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39416/0 (socket says 172.21.15.116:39416) 2023-11-08T08:18:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.757+0000 7fc0ad572700 1 -- 172.21.15.116:0/2063281095 learned_addr learned my addr 172.21.15.116:0/2063281095 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.757+0000 7fc0acd71700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc0ad572700 1 -- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a80cbc50 msgr2=0x7fc0a80c4a10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc0ad572700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a80cbc50 0x7fc0a80c4a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc0ad572700 1 -- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a80b0c50 msgr2=0x7fc0a80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc0ad572700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80703e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc0ad572700 1 -- 172.21.15.116:0/2063281095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a0009a70 con 0x7fc0a806d230 2023-11-08T08:18:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efcf4029460 con 0x7efd0408a5d0 2023-11-08T08:18:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc0ad572700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 0x7fc0a80c7a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4001eb0 tx=0x7fc0a400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18003e60 con 0x7fbd240718e0 2023-11-08T08:18:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd18011630 con 0x7fbd240718e0 2023-11-08T08:18:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd180119e0 con 0x7fbd240718e0 2023-11-08T08:18:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40054e0 con 0x7fc0a806d230 2023-11-08T08:18:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4003680 con 0x7fc0a806d230 2023-11-08T08:18:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40039f0 con 0x7fc0a806d230 2023-11-08T08:18:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7efcfb7fe700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efce40682f0 0x7efce406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.759+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efcf4088cc0 con 0x7efd0408a5d0 2023-11-08T08:18:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.757+0000 7fa504e51700 1 Processor -- start 2023-11-08T08:18:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.758+0000 7fa504e51700 1 -- start start 2023-11-08T08:18:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa504e51700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa50006b770 0x7fa5000adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa504e51700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa5000b0540 0x7fa5000b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa504e51700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5000ce1d0 con 0x7fa50008a5d0 2023-11-08T08:18:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa504e51700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5000cc690 con 0x7fa50006b770 2023-11-08T08:18:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa504e51700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000b0ec0 con 0x7fa5000b0540 2023-11-08T08:18:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa50006b770 0x7fa5000adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa50006b770 0x7fa5000adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39420/0 (socket says 172.21.15.116:39420) 2023-11-08T08:18:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 -- 172.21.15.116:0/142324156 learned_addr learned my addr 172.21.15.116:0/142324156 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fed9d700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa5000b0540 0x7fa5000b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.759+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efcf40553d0 con 0x7efd0408a5d0 2023-11-08T08:18:50.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7efcfbfff700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efce40682f0 0x7efce406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 -- 172.21.15.116:0/142324156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa5000b0540 msgr2=0x7fa5000b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa5000b0540 0x7fa5000b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 -- 172.21.15.116:0/142324156 >> v1:172.21.15.116:6789/0 conn(0x7fa50008a5d0 legacy=0x7fa5000cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:50.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fdd9b700 1 -- 172.21.15.116:0/142324156 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5000ce410 con 0x7fa50006b770 2023-11-08T08:18:50.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.760+0000 7fa4fed9d700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa5000b0540 0x7fa5000b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:50.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7fa4fdd9b700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa5000adf60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00098f0 tx=0x7fa4f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd33d613b78bf3a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.759+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a80cb090 con 0x7fc0a806d230 2023-11-08T08:18:50.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.759+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a80cadb0 con 0x7fc0a806d230 2023-11-08T08:18:50.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7efcfbfff700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efce40682f0 0x7efce406a7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efcec000f60 tx=0x7efcec007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240ceb20 con 0x7fbd240718e0 2023-11-08T08:18:50.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24161010 con 0x7fbd240718e0 2023-11-08T08:18:50.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7fa4fd59a700 1 -- 172.21.15.116:0/142324156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0004230 con 0x7fa50006b770 2023-11-08T08:18:50.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7fa4fd59a700 1 -- 172.21.15.116:0/142324156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f0004390 con 0x7fa50006b770 2023-11-08T08:18:50.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.761+0000 7fa4fd59a700 1 -- 172.21.15.116:0/142324156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f00036a0 con 0x7fa50006b770 2023-11-08T08:18:50.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.762+0000 7fa504e51700 1 -- 172.21.15.116:0/142324156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 msgr2=0x7fa5000adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.762+0000 7fa504e51700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa5000adf60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00098f0 tx=0x7fa4f0009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:50.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.763+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd240ce740 con 0x7fbd240718e0 2023-11-08T08:18:50.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.763+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a80adba0 con 0x7fc0a806d230 2023-11-08T08:18:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.764+0000 7fa504e51700 1 -- 172.21.15.116:0/142324156 shutdown_connections 2023-11-08T08:18:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.764+0000 7fa504e51700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa5000b0540 0x7fa5000b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.764+0000 7fa504e51700 1 --2- 172.21.15.116:0/142324156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa5000adf60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.764+0000 7fa504e51700 1 -- 172.21.15.116:0/142324156 >> 172.21.15.116:0/142324156 conn(0x7fa50006aed0 msgr2=0x7fa5000cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:50.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.765+0000 7fa504e51700 1 -- 172.21.15.116:0/142324156 shutdown_connections 2023-11-08T08:18:50.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.765+0000 7fa504e51700 1 -- 172.21.15.116:0/142324156 wait complete. 2023-11-08T08:18:50.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.767+0000 7fa504e51700 1 Processor -- start 2023-11-08T08:18:50.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.767+0000 7fa504e51700 1 -- start start 2023-11-08T08:18:50.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.768+0000 7fa504e51700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa500088520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.768+0000 7fa504e51700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa50008a5d0 0x7fa500088a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.768+0000 7fa504e51700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5000b0540 0x7fa50006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.768+0000 7fa504e51700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000c5f90 con 0x7fa50006b770 2023-11-08T08:18:50.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.768+0000 7fa504e51700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000c6270 con 0x7fa5000b0540 2023-11-08T08:18:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.768+0000 7fa504e51700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000c6550 con 0x7fa50008a5d0 2023-11-08T08:18:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa500088520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa500088520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39432/0 (socket says 172.21.15.116:39432) 2023-11-08T08:18:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 -- 172.21.15.116:0/2661388361 learned_addr learned my addr 172.21.15.116:0/2661388361 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fed9d700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5000b0540 0x7fa50006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 -- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa50008a5d0 msgr2=0x7fa500088a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:18:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa50008a5d0 0x7fa500088a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 -- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5000b0540 msgr2=0x7fa50006fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5000b0540 0x7fa50006fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.769+0000 7fa4fe59c700 1 -- 172.21.15.116:0/2661388361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f0009550 con 0x7fa50006b770 2023-11-08T08:18:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.770+0000 7fa4fe59c700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa500088520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c4a0 tx=0x7fa4e800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.770+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd18017020 con 0x7fbd240718e0 2023-11-08T08:18:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.770+0000 7fbd12ffd700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbd040683c0 0x7fbd0406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.770+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbd18087480 con 0x7fbd240718e0 2023-11-08T08:18:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.772+0000 7fbd2259c700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbd040683c0 0x7fbd0406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.773+0000 7fbd2259c700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbd040683c0 0x7fbd0406a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009ac0 tx=0x7fbd140091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.772+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc0a4019030 con 0x7fc0a806d230 2023-11-08T08:18:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.772+0000 7fc09dffb700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc08c0682f0 0x7fc08c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.772+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc0a4084f00 con 0x7fc0a806d230 2023-11-08T08:18:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.773+0000 7fc0acd71700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc08c0682f0 0x7fc08c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.773+0000 7fc0acd71700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc08c0682f0 0x7fc08c06a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc0a806cde0 tx=0x7fc098009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.774+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd18053c70 con 0x7fbd240718e0 2023-11-08T08:18:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.774+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e80054e0 con 0x7fa50006b770 2023-11-08T08:18:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.774+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5000cb190 con 0x7fa50006b770 2023-11-08T08:18:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.774+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5000caeb0 con 0x7fa50006b770 2023-11-08T08:18:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.776+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5000cbfa0 con 0x7fa50006b770 2023-11-08T08:18:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.792+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc0a404b5e0 con 0x7fc0a806d230 2023-11-08T08:18:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.795+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e8003680 con 0x7fa50006b770 2023-11-08T08:18:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.795+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e80039f0 con 0x7fa50006b770 2023-11-08T08:18:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.795+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa4e8019030 con 0x7fa50006b770 2023-11-08T08:18:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.795+0000 7fa4f6ffd700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4e0068370 0x7fa4e006a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.796+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa4e8085860 con 0x7fa50006b770 2023-11-08T08:18:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.796+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4e80b0680 con 0x7fa50006b770 2023-11-08T08:18:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.796+0000 7fa4fdd9b700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4e0068370 0x7fa4e006a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:50.797+0000 7fa4fdd9b700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4e0068370 0x7fa4e006a820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5000c83c0 tx=0x7fa4f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.072+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0bf8005870 con 0x7f0c1006c930 2023-11-08T08:18:51.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.076+0000 7f0c067fc700 1 -- 172.21.15.116:0/1345035584 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c00016090 con 0x7f0c1006c930 2023-11-08T08:18:51.082 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-08T08:18:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bfc068520 msgr2=0x7f0bfc06a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bfc068520 0x7f0bfc06a9d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0c101675f0 tx=0x7f0c08009040 comp rx=0 tx=0).stop 2023-11-08T08:18:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 msgr2=0x7f0c100b18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 0x7f0c100b18f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0c10067450 tx=0x7f0c00004750 comp rx=0 tx=0).stop 2023-11-08T08:18:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 shutdown_connections 2023-11-08T08:18:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bfc068520 0x7f0bfc06a9d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c100810f0 0x7f0c10081840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1006c930 0x7f0c100b18f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 --2- 172.21.15.116:0/1345035584 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0c100ae020 0x7f0c100b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.080+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 >> 172.21.15.116:0/1345035584 conn(0x7f0c1006aed0 msgr2=0x7f0c10168120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.082+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 shutdown_connections 2023-11-08T08:18:51.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.082+0000 7f0c17bca700 1 -- 172.21.15.116:0/1345035584 wait complete. 2023-11-08T08:18:51.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2023-11-08T08:18:51.100 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.102+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f17b80743f0 con 0x7f17b80b0ef0 2023-11-08T08:18:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.102+0000 7f179ffff700 1 -- 172.21.15.116:0/642282618 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f17a8017020 con 0x7f17b80b0ef0 2023-11-08T08:18:51.105 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.106+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17a40683c0 msgr2=0x7f17a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.106+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17a40683c0 0x7f17a406a870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f17b80b1630 tx=0x7f17ac008030 comp rx=0 tx=0).stop 2023-11-08T08:18:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.106+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 msgr2=0x7f17b80af570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.106+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 0x7f17b80af570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17a800c4d0 tx=0x7f17a800c7e0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 shutdown_connections 2023-11-08T08:18:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b80afab0 0x7f17b8202de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17a40683c0 0x7f17a406a870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b80adda0 0x7f17b8202200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 --2- 172.21.15.116:0/642282618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17b80b0ef0 0x7f17b80af570 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 >> 172.21.15.116:0/642282618 conn(0x7f17b80cb540 msgr2=0x7f17b80ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 shutdown_connections 2023-11-08T08:18:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.107+0000 7f17bdbf9700 1 -- 172.21.15.116:0/642282618 wait complete. 2023-11-08T08:18:51.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-11-08T08:18:51.124 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.129+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7efd040743f0 con 0x7efd0408a5d0 2023-11-08T08:18:51.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.130+0000 7efcfb7fe700 1 -- 172.21.15.116:0/3663231206 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efcf40589f0 con 0x7efd0408a5d0 2023-11-08T08:18:51.131 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:51.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.131+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa7a40055a0 con 0x7fa7bc0cf710 2023-11-08T08:18:51.132 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:51.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.132+0000 7fa7b27fc700 1 -- 172.21.15.116:0/3818642238 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7b4017090 con 0x7fa7bc0cf710 2023-11-08T08:18:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efce40682f0 msgr2=0x7efce406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efce40682f0 0x7efce406a7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efcec000f60 tx=0x7efcec007a40 comp rx=0 tx=0).stop 2023-11-08T08:18:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 msgr2=0x7efd04089580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 0x7efd04089580 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efcf4000c00 tx=0x7efcf4004750 comp rx=0 tx=0).stop 2023-11-08T08:18:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 shutdown_connections 2023-11-08T08:18:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd040c3c10 0x7efd040c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efd0408a5d0 0x7efd04089580 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efce40682f0 0x7efce406a7a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 --2- 172.21.15.116:0/3663231206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cac10 0x7efd040c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 >> 172.21.15.116:0/3663231206 conn(0x7efd0406aed0 msgr2=0x7efd04071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 shutdown_connections 2023-11-08T08:18:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7efd0958a700 1 -- 172.21.15.116:0/3663231206 wait complete. 2023-11-08T08:18:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7a00683c0 msgr2=0x7fa7a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7a00683c0 0x7fa7a006a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009ac0 tx=0x7fa7ac0091c0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc0cf710 msgr2=0x7fa7bc15e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc0cf710 0x7fa7bc15e3f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc06b0f0 tx=0x7fa7b4004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 shutdown_connections 2023-11-08T08:18:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7a00683c0 0x7fa7a006a870 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009ac0 tx=0x7fa7ac0091c0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7bc0cf710 0x7fa7bc15e3f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc06a980 0x7fa7bc15b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 --2- 172.21.15.116:0/3818642238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7bc071890 0x7fa7bc0b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.135+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 >> 172.21.15.116:0/3818642238 conn(0x7fa7bc06c010 msgr2=0x7fa7bc0b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.138+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 shutdown_connections 2023-11-08T08:18:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.138+0000 7fa7c40d8700 1 -- 172.21.15.116:0/3818642238 wait complete. 2023-11-08T08:18:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.140+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fbd240743f0 con 0x7fbd240718e0 2023-11-08T08:18:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.141+0000 7fbd12ffd700 1 -- 172.21.15.116:0/901148030 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd18057290 con 0x7fbd240718e0 2023-11-08T08:18:51.142 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-08T08:18:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.144+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbd040683c0 msgr2=0x7fbd0406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.144+0000 7fbd28853700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbd040683c0 0x7fbd0406a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009ac0 tx=0x7fbd140091c0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.144+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd240718e0 msgr2=0x7fbd240b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.144+0000 7fbd28853700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd240718e0 0x7fbd240b50f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbd18015000 tx=0x7fbd18004750 comp rx=0 tx=0).stop 2023-11-08T08:18:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f35ec0055a0 con 0x7f360c1652c0 2023-11-08T08:18:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 shutdown_connections 2023-11-08T08:18:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbd040683c0 0x7fbd0406a870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd2415c810 0x7fbd240aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbd2406cdc0 0x7fbd241554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 --2- 172.21.15.116:0/901148030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd240718e0 0x7fbd240b50f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 >> 172.21.15.116:0/901148030 conn(0x7fbd2406c010 msgr2=0x7fbd2406d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7f3608ff9700 1 -- 172.21.15.116:0/3122759409 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3600057390 con 0x7f360c1652c0 2023-11-08T08:18:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.146+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 shutdown_connections 2023-11-08T08:18:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.147+0000 7fbd28853700 1 -- 172.21.15.116:0/901148030 wait complete. 2023-11-08T08:18:51.150 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:51.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35f80682f0 msgr2=0x7f35f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35f80682f0 0x7f35f806a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35fc007fd0 tx=0x7f35fc009040 comp rx=0 tx=0).stop 2023-11-08T08:18:51.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 msgr2=0x7f360c161350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 0x7f360c161350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3600009c40 tx=0x7f3600004750 comp rx=0 tx=0).stop 2023-11-08T08:18:51.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 shutdown_connections 2023-11-08T08:18:51.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35f80682f0 0x7f35f806a7a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f360c15be20 0x7f360c0b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f360c160ac0 0x7f360c1f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.150+0000 7f36129d1700 1 --2- 172.21.15.116:0/3122759409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360c1652c0 0x7f360c161350 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.151+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 >> 172.21.15.116:0/3122759409 conn(0x7f360c06c520 msgr2=0x7f360c0b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.151+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 shutdown_connections 2023-11-08T08:18:51.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.151+0000 7f36129d1700 1 -- 172.21.15.116:0/3122759409 wait complete. 2023-11-08T08:18:51.155 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-11-08T08:18:51.156 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2023-11-08T08:18:51.157 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2023-11-08T08:18:51.166 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-11-08T08:18:51.167 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.180+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc0a80adf60 con 0x7fc0a806d230 2023-11-08T08:18:51.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.181+0000 7fc09dffb700 1 -- 172.21.15.116:0/2063281095 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0a4017020 con 0x7fc0a806d230 2023-11-08T08:18:51.181 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:51.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc08c0682f0 msgr2=0x7fc08c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc08c0682f0 0x7fc08c06a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc0a806cde0 tx=0x7fc098009040 comp rx=0 tx=0).stop 2023-11-08T08:18:51.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 msgr2=0x7fc0a80c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 0x7fc0a80c7a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4001eb0 tx=0x7fc0a400c800 comp rx=0 tx=0).stop 2023-11-08T08:18:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 shutdown_connections 2023-11-08T08:18:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a80cbc50 0x7fc0a80c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a806d230 0x7fc0a80c7a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc08c0682f0 0x7fc08c06a7a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 --2- 172.21.15.116:0/2063281095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a80b0c50 0x7fc0a80703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 >> 172.21.15.116:0/2063281095 conn(0x7fc0a806aed0 msgr2=0x7fc0a80cc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 shutdown_connections 2023-11-08T08:18:51.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.184+0000 7fc0af7d6700 1 -- 172.21.15.116:0/2063281095 wait complete. 2023-11-08T08:18:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.185+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa50008ac20 con 0x7fa50006b770 2023-11-08T08:18:51.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.186+0000 7fa4f6ffd700 1 -- 172.21.15.116:0/2661388361 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa4e8021090 con 0x7fa50006b770 2023-11-08T08:18:51.187 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-08T08:18:51.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4e0068370 msgr2=0x7fa4e006a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4e0068370 0x7fa4e006a820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5000c83c0 tx=0x7fa4f0008030 comp rx=0 tx=0).stop 2023-11-08T08:18:51.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 msgr2=0x7fa500088520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa500088520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800c4a0 tx=0x7fa4e800c7b0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 shutdown_connections 2023-11-08T08:18:51.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4e0068370 0x7fa4e006a820 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5000b0540 0x7fa50006fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa50006b770 0x7fa500088520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 --2- 172.21.15.116:0/2661388361 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa50008a5d0 0x7fa500088a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 >> 172.21.15.116:0/2661388361 conn(0x7fa50006aed0 msgr2=0x7fa5000af4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 shutdown_connections 2023-11-08T08:18:51.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.189+0000 7fa504e51700 1 -- 172.21.15.116:0/2661388361 wait complete. 2023-11-08T08:18:51.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-11-08T08:18:51.200 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.203 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2023-11-08T08:18:51.203 DEBUG:teuthology.parallel:result is None 2023-11-08T08:18:51.204 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-08T08:18:51.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:18:51.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.364+0000 7fc997c9a700 1 Processor -- start 2023-11-08T08:18:51.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.364+0000 7fc997c9a700 1 -- start start 2023-11-08T08:18:51.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc997c9a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc99006e400 0x7fc9900b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc997c9a700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc990160e70 0x7fc99015c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc997c9a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99006e900 con 0x7fc9900d0c60 2023-11-08T08:18:51.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc997c9a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc990160390 con 0x7fc99006e400 2023-11-08T08:18:51.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc997c9a700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc990160670 con 0x7fc990160e70 2023-11-08T08:18:51.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc99006e400 0x7fc9900b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc99006e400 0x7fc9900b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39436/0 (socket says 172.21.15.116:39436) 2023-11-08T08:18:51.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 -- 172.21.15.116:0/3823694561 learned_addr learned my addr 172.21.15.116:0/3823694561 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:51.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc996237700 1 --2- 172.21.15.116:0/3823694561 >> v2:172.21.15.129:3300/0 conn(0x7fc990160e70 0x7fc99015c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 -- 172.21.15.116:0/3823694561 >> v2:172.21.15.129:3300/0 conn(0x7fc990160e70 msgr2=0x7fc99015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:51.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 --2- 172.21.15.116:0/3823694561 >> v2:172.21.15.129:3300/0 conn(0x7fc990160e70 0x7fc99015c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 -- 172.21.15.116:0/3823694561 >> v1:172.21.15.129:6789/0 conn(0x7fc9900d0c60 legacy=0x7fc9900d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:51.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 -- 172.21.15.116:0/3823694561 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc990160950 con 0x7fc99006e400 2023-11-08T08:18:51.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc996237700 1 --2- 172.21.15.116:0/3823694561 >> v2:172.21.15.129:3300/0 conn(0x7fc990160e70 0x7fc99015c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:51.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc995235700 1 --2- 172.21.15.116:0/3823694561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b5be0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc990068250 tx=0x7fc980009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b16e66eac89d6f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc994a34700 1 -- 172.21.15.116:0/3823694561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980004230 con 0x7fc99006e400 2023-11-08T08:18:51.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc994a34700 1 -- 172.21.15.116:0/3823694561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980004390 con 0x7fc99006e400 2023-11-08T08:18:51.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.365+0000 7fc994a34700 1 -- 172.21.15.116:0/3823694561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9800036a0 con 0x7fc99006e400 2023-11-08T08:18:51.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 -- 172.21.15.116:0/3823694561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 msgr2=0x7fc9900b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3823694561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b5be0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc990068250 tx=0x7fc980009bd0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 -- 172.21.15.116:0/3823694561 shutdown_connections 2023-11-08T08:18:51.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3823694561 >> v2:172.21.15.129:3300/0 conn(0x7fc990160e70 0x7fc99015c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3823694561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b5be0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 -- 172.21.15.116:0/3823694561 >> 172.21.15.116:0/3823694561 conn(0x7fc99006a270 msgr2=0x7fc99006a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 -- 172.21.15.116:0/3823694561 shutdown_connections 2023-11-08T08:18:51.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.366+0000 7fc997c9a700 1 -- 172.21.15.116:0/3823694561 wait complete. 2023-11-08T08:18:51.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.367+0000 7fc997c9a700 1 Processor -- start 2023-11-08T08:18:51.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.367+0000 7fc997c9a700 1 -- start start 2023-11-08T08:18:51.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc997c9a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc997c9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9900d0c60 0x7fc99015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc997c9a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc990160e70 0x7fc99015bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc997c9a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900b08d0 con 0x7fc99006e400 2023-11-08T08:18:51.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc997c9a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900b0b70 con 0x7fc9900d0c60 2023-11-08T08:18:51.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc997c9a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900b0e50 con 0x7fc990160e70 2023-11-08T08:18:51.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39442/0 (socket says 172.21.15.116:39442) 2023-11-08T08:18:51.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 -- 172.21.15.116:0/3163211559 learned_addr learned my addr 172.21.15.116:0/3163211559 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:51.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 -- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc990160e70 msgr2=0x7fc99015bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc996237700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc990160e70 0x7fc99015bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995235700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9900d0c60 0x7fc99015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc990160e70 0x7fc99015bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 -- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9900d0c60 msgr2=0x7fc99015b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9900d0c60 0x7fc99015b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 -- 172.21.15.116:0/3163211559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980009550 con 0x7fc99006e400 2023-11-08T08:18:51.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.368+0000 7fc995a36700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b2bb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c4d0 tx=0x7fc98c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.369+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c011790 con 0x7fc99006e400 2023-11-08T08:18:51.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.369+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c0118f0 con 0x7fc99006e400 2023-11-08T08:18:51.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.369+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99015ceb0 con 0x7fc99006e400 2023-11-08T08:18:51.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.369+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c011ca0 con 0x7fc99006e400 2023-11-08T08:18:51.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.369+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9900d0740 con 0x7fc99006e400 2023-11-08T08:18:51.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.370+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99015d040 con 0x7fc99006e400 2023-11-08T08:18:51.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.370+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc98c010050 con 0x7fc99006e400 2023-11-08T08:18:51.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.370+0000 7fc9867fc700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc9740682f0 0x7fc97406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.371+0000 7fc995235700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc9740682f0 0x7fc97406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.371+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc98c085970 con 0x7fc99006e400 2023-11-08T08:18:51.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.371+0000 7fc995235700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc9740682f0 0x7fc97406a7a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc99006e830 tx=0x7fc980008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.375+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc98c0521d0 con 0x7fc99006e400 2023-11-08T08:18:51.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.550+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc990154150 con 0x7fc9740682f0 2023-11-08T08:18:51.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.558+0000 7fc9867fc700 1 -- 172.21.15.116:0/3163211559 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286821 (secure 0 0 0) 0x7fc990154150 con 0x7fc9740682f0 2023-11-08T08:18:51.562 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:51.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc9740682f0 msgr2=0x7fc97406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc9740682f0 0x7fc97406a7a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc99006e830 tx=0x7fc980008030 comp rx=0 tx=0).stop 2023-11-08T08:18:51.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 msgr2=0x7fc9900b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b2bb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c4d0 tx=0x7fc98c00c890 comp rx=0 tx=0).stop 2023-11-08T08:18:51.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 shutdown_connections 2023-11-08T08:18:51.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc990160e70 0x7fc99015bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99006e400 0x7fc9900b2bb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc9740682f0 0x7fc97406a7a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 --2- 172.21.15.116:0/3163211559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9900d0c60 0x7fc99015b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 >> 172.21.15.116:0/3163211559 conn(0x7fc99006a270 msgr2=0x7fc99006f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 shutdown_connections 2023-11-08T08:18:51.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.561+0000 7fc997c9a700 1 -- 172.21.15.116:0/3163211559 wait complete. 2023-11-08T08:18:51.566 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:18:51.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-11-08T08:18:50.353383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65796,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":754908924,"statfs":{"total":773094113280,"available":773026738176,"internally_reserved":0,"allocated":2887680,"data_stored":1779623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.867573"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899181+0000","last_change":"2023-11-08T08:18:42.907639+0000","last_active":"2023-11-08T08:18:43.899181+0000","last_peered":"2023-11-08T08:18:43.899181+0000","last_clean":"2023-11-08T08:18:43.899181+0000","last_became_active":"2023-11-08T08:18:42.906449+0000","last_became_peered":"2023-11-08T08:18:42.906449+0000","last_unstale":"2023-11-08T08:18:43.899181+0000","last_undegraded":"2023-11-08T08:18:43.899181+0000","last_fullsized":"2023-11-08T08:18:43.899181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893783+0000","last_change":"2023-11-08T08:18:42.912729+0000","last_active":"2023-11-08T08:18:43.893783+0000","last_peered":"2023-11-08T08:18:43.893783+0000","last_clean":"2023-11-08T08:18:43.893783+0000","last_became_active":"2023-11-08T08:18:42.912482+0000","last_became_peered":"2023-11-08T08:18:42.912482+0000","last_unstale":"2023-11-08T08:18:43.893783+0000","last_undegraded":"2023-11-08T08:18:43.893783+0000","last_fullsized":"2023-11-08T08:18:43.893783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272081+0000","last_change":"2023-11-08T08:18:42.904767+0000","last_active":"2023-11-08T08:18:44.272081+0000","last_peered":"2023-11-08T08:18:44.272081+0000","last_clean":"2023-11-08T08:18:44.272081+0000","last_became_active":"2023-11-08T08:18:42.904512+0000","last_became_peered":"2023-11-08T08:18:42.904512+0000","last_unstale":"2023-11-08T08:18:44.272081+0000","last_undegraded":"2023-11-08T08:18:44.272081+0000","last_fullsized":"2023-11-08T08:18:44.272081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271750+0000","last_change":"2023-11-08T08:18:42.901716+0000","last_active":"2023-11-08T08:18:44.271750+0000","last_peered":"2023-11-08T08:18:44.271750+0000","last_clean":"2023-11-08T08:18:44.271750+0000","last_became_active":"2023-11-08T08:18:42.901477+0000","last_became_peered":"2023-11-08T08:18:42.901477+0000","last_unstale":"2023-11-08T08:18:44.271750+0000","last_undegraded":"2023-11-08T08:18:44.271750+0000","last_fullsized":"2023-11-08T08:18:44.271750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272162+0000","last_change":"2023-11-08T08:18:42.906339+0000","last_active":"2023-11-08T08:18:44.272162+0000","last_peered":"2023-11-08T08:18:44.272162+0000","last_clean":"2023-11-08T08:18:44.272162+0000","last_became_active":"2023-11-08T08:18:42.906082+0000","last_became_peered":"2023-11-08T08:18:42.906082+0000","last_unstale":"2023-11-08T08:18:44.272162+0000","last_undegraded":"2023-11-08T08:18:44.272162+0000","last_fullsized":"2023-11-08T08:18:44.272162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272523+0000","last_change":"2023-11-08T08:18:42.907735+0000","last_active":"2023-11-08T08:18:44.272523+0000","last_peered":"2023-11-08T08:18:44.272523+0000","last_clean":"2023-11-08T08:18:44.272523+0000","last_became_active":"2023-11-08T08:18:42.907414+0000","last_became_peered":"2023-11-08T08:18:42.907414+0000","last_unstale":"2023-11-08T08:18:44.272523+0000","last_undegraded":"2023-11-08T08:18:44.272523+0000","last_fullsized":"2023-11-08T08:18:44.272523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271898+0000","last_change":"2023-11-08T08:18:42.904968+0000","last_active":"2023-11-08T08:18:44.271898+0000","last_peered":"2023-11-08T08:18:44.271898+0000","last_clean":"2023-11-08T08:18:44.271898+0000","last_became_active":"2023-11-08T08:18:42.904586+0000","last_became_peered":"2023-11-08T08:18:42.904586+0000","last_unstale":"2023-11-08T08:18:44.271898+0000","last_undegraded":"2023-11-08T08:18:44.271898+0000","last_fullsized":"2023-11-08T08:18:44.271898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:08.733824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898246+0000","last_change":"2023-11-08T08:18:42.898509+0000","last_active":"2023-11-08T08:18:43.898246+0000","last_peered":"2023-11-08T08:18:43.898246+0000","last_clean":"2023-11-08T08:18:43.898246+0000","last_became_active":"2023-11-08T08:18:42.898230+0000","last_became_peered":"2023-11-08T08:18:42.898230+0000","last_unstale":"2023-11-08T08:18:43.898246+0000","last_undegraded":"2023-11-08T08:18:43.898246+0000","last_fullsized":"2023-11-08T08:18:43.898246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273075+0000","last_change":"2023-11-08T08:18:42.906809+0000","last_active":"2023-11-08T08:18:44.273075+0000","last_peered":"2023-11-08T08:18:44.273075+0000","last_clean":"2023-11-08T08:18:44.273075+0000","last_became_active":"2023-11-08T08:18:42.906517+0000","last_became_peered":"2023-11-08T08:18:42.906517+0000","last_unstale":"2023-11-08T08:18:44.273075+0000","last_undegraded":"2023-11-08T08:18:44.273075+0000","last_fullsized":"2023-11-08T08:18:44.273075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898585+0000","last_change":"2023-11-08T08:18:42.906651+0000","last_active":"2023-11-08T08:18:43.898585+0000","last_peered":"2023-11-08T08:18:43.898585+0000","last_clean":"2023-11-08T08:18:43.898585+0000","last_became_active":"2023-11-08T08:18:42.906379+0000","last_became_peered":"2023-11-08T08:18:42.906379+0000","last_unstale":"2023-11-08T08:18:43.898585+0000","last_undegraded":"2023-11-08T08:18:43.898585+0000","last_fullsized":"2023-11-08T08:18:43.898585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898449+0000","last_change":"2023-11-08T08:18:42.898832+0000","last_active":"2023-11-08T08:18:43.898449+0000","last_peered":"2023-11-08T08:18:43.898449+0000","last_clean":"2023-11-08T08:18:43.898449+0000","last_became_active":"2023-11-08T08:18:42.898581+0000","last_became_peered":"2023-11-08T08:18:42.898581+0000","last_unstale":"2023-11-08T08:18:43.898449+0000","last_undegraded":"2023-11-08T08:18:43.898449+0000","last_fullsized":"2023-11-08T08:18:43.898449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898948+0000","last_change":"2023-11-08T08:18:42.911043+0000","last_active":"2023-11-08T08:18:43.898948+0000","last_peered":"2023-11-08T08:18:43.898948+0000","last_clean":"2023-11-08T08:18:43.898948+0000","last_became_active":"2023-11-08T08:18:42.910756+0000","last_became_peered":"2023-11-08T08:18:42.910756+0000","last_unstale":"2023-11-08T08:18:43.898948+0000","last_undegraded":"2023-11-08T08:18:43.898948+0000","last_fullsized":"2023-11-08T08:18:43.898948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272630+0000","last_change":"2023-11-08T08:18:42.915118+0000","last_active":"2023-11-08T08:18:44.272630+0000","last_peered":"2023-11-08T08:18:44.272630+0000","last_clean":"2023-11-08T08:18:44.272630+0000","last_became_active":"2023-11-08T08:18:42.914854+0000","last_became_peered":"2023-11-08T08:18:42.914854+0000","last_unstale":"2023-11-08T08:18:44.272630+0000","last_undegraded":"2023-11-08T08:18:44.272630+0000","last_fullsized":"2023-11-08T08:18:44.272630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898604+0000","last_change":"2023-11-08T08:18:42.900369+0000","last_active":"2023-11-08T08:18:43.898604+0000","last_peered":"2023-11-08T08:18:43.898604+0000","last_clean":"2023-11-08T08:18:43.898604+0000","last_became_active":"2023-11-08T08:18:42.900121+0000","last_became_peered":"2023-11-08T08:18:42.900121+0000","last_unstale":"2023-11-08T08:18:43.898604+0000","last_undegraded":"2023-11-08T08:18:43.898604+0000","last_fullsized":"2023-11-08T08:18:43.898604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271716+0000","last_change":"2023-11-08T08:18:42.912567+0000","last_active":"2023-11-08T08:18:44.271716+0000","last_peered":"2023-11-08T08:18:44.271716+0000","last_clean":"2023-11-08T08:18:44.271716+0000","last_became_active":"2023-11-08T08:18:42.912299+0000","last_became_peered":"2023-11-08T08:18:42.912299+0000","last_unstale":"2023-11-08T08:18:44.271716+0000","last_undegraded":"2023-11-08T08:18:44.271716+0000","last_fullsized":"2023-11-08T08:18:44.271716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271758+0000","last_change":"2023-11-08T08:18:42.902375+0000","last_active":"2023-11-08T08:18:44.271758+0000","last_peered":"2023-11-08T08:18:44.271758+0000","last_clean":"2023-11-08T08:18:44.271758+0000","last_became_active":"2023-11-08T08:18:42.901640+0000","last_became_peered":"2023-11-08T08:18:42.901640+0000","last_unstale":"2023-11-08T08:18:44.271758+0000","last_undegraded":"2023-11-08T08:18:44.271758+0000","last_fullsized":"2023-11-08T08:18:44.271758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272211+0000","last_change":"2023-11-08T08:18:42.905755+0000","last_active":"2023-11-08T08:18:44.272211+0000","last_peered":"2023-11-08T08:18:44.272211+0000","last_clean":"2023-11-08T08:18:44.272211+0000","last_became_active":"2023-11-08T08:18:42.905498+0000","last_became_peered":"2023-11-08T08:18:42.905498+0000","last_unstale":"2023-11-08T08:18:44.272211+0000","last_undegraded":"2023-11-08T08:18:44.272211+0000","last_fullsized":"2023-11-08T08:18:44.272211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272018+0000","last_change":"2023-11-08T08:18:42.900992+0000","last_active":"2023-11-08T08:18:44.272018+0000","last_peered":"2023-11-08T08:18:44.272018+0000","last_clean":"2023-11-08T08:18:44.272018+0000","last_became_active":"2023-11-08T08:18:42.900747+0000","last_became_peered":"2023-11-08T08:18:42.900747+0000","last_unstale":"2023-11-08T08:18:44.272018+0000","last_undegraded":"2023-11-08T08:18:44.272018+0000","last_fullsized":"2023-11-08T08:18:44.272018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899009+0000","last_change":"2023-11-08T08:18:42.910975+0000","last_active":"2023-11-08T08:18:43.899009+0000","last_peered":"2023-11-08T08:18:43.899009+0000","last_clean":"2023-11-08T08:18:43.899009+0000","last_became_active":"2023-11-08T08:18:42.910701+0000","last_became_peered":"2023-11-08T08:18:42.910701+0000","last_unstale":"2023-11-08T08:18:43.899009+0000","last_undegraded":"2023-11-08T08:18:43.899009+0000","last_fullsized":"2023-11-08T08:18:43.899009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898283+0000","last_change":"2023-11-08T08:18:42.912881+0000","last_active":"2023-11-08T08:18:43.898283+0000","last_peered":"2023-11-08T08:18:43.898283+0000","last_clean":"2023-11-08T08:18:43.898283+0000","last_became_active":"2023-11-08T08:18:42.912660+0000","last_became_peered":"2023-11-08T08:18:42.912660+0000","last_unstale":"2023-11-08T08:18:43.898283+0000","last_undegraded":"2023-11-08T08:18:43.898283+0000","last_fullsized":"2023-11-08T08:18:43.898283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898341+0000","last_change":"2023-11-08T08:18:42.901149+0000","last_active":"2023-11-08T08:18:43.898341+0000","last_peered":"2023-11-08T08:18:43.898341+0000","last_clean":"2023-11-08T08:18:43.898341+0000","last_became_active":"2023-11-08T08:18:42.900728+0000","last_became_peered":"2023-11-08T08:18:42.900728+0000","last_unstale":"2023-11-08T08:18:43.898341+0000","last_undegraded":"2023-11-08T08:18:43.898341+0000","last_fullsized":"2023-11-08T08:18:43.898341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272020+0000","last_change":"2023-11-08T08:18:42.911764+0000","last_active":"2023-11-08T08:18:44.272020+0000","last_peered":"2023-11-08T08:18:44.272020+0000","last_clean":"2023-11-08T08:18:44.272020+0000","last_became_active":"2023-11-08T08:18:42.911477+0000","last_became_peered":"2023-11-08T08:18:42.911477+0000","last_unstale":"2023-11-08T08:18:44.272020+0000","last_undegraded":"2023-11-08T08:18:44.272020+0000","last_fullsized":"2023-11-08T08:18:44.272020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272570+0000","last_change":"2023-11-08T08:18:42.909892+0000","last_active":"2023-11-08T08:18:44.272570+0000","last_peered":"2023-11-08T08:18:44.272570+0000","last_clean":"2023-11-08T08:18:44.272570+0000","last_became_active":"2023-11-08T08:18:42.909646+0000","last_became_peered":"2023-11-08T08:18:42.909646+0000","last_unstale":"2023-11-08T08:18:44.272570+0000","last_undegraded":"2023-11-08T08:18:44.272570+0000","last_fullsized":"2023-11-08T08:18:44.272570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898218+0000","last_change":"2023-11-08T08:18:42.897278+0000","last_active":"2023-11-08T08:18:43.898218+0000","last_peered":"2023-11-08T08:18:43.898218+0000","last_clean":"2023-11-08T08:18:43.898218+0000","last_became_active":"2023-11-08T08:18:42.896923+0000","last_became_peered":"2023-11-08T08:18:42.896923+0000","last_unstale":"2023-11-08T08:18:43.898218+0000","last_undegraded":"2023-11-08T08:18:43.898218+0000","last_fullsized":"2023-11-08T08:18:43.898218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273144+0000","last_change":"2023-11-08T08:18:42.909514+0000","last_active":"2023-11-08T08:18:44.273144+0000","last_peered":"2023-11-08T08:18:44.273144+0000","last_clean":"2023-11-08T08:18:44.273144+0000","last_became_active":"2023-11-08T08:18:42.909246+0000","last_became_peered":"2023-11-08T08:18:42.909246+0000","last_unstale":"2023-11-08T08:18:44.273144+0000","last_undegraded":"2023-11-08T08:18:44.273144+0000","last_fullsized":"2023-11-08T08:18:44.273144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272924+0000","last_change":"2023-11-08T08:18:42.912860+0000","last_active":"2023-11-08T08:18:44.272924+0000","last_peered":"2023-11-08T08:18:44.272924+0000","last_clean":"2023-11-08T08:18:44.272924+0000","last_became_active":"2023-11-08T08:18:42.912613+0000","last_became_peered":"2023-11-08T08:18:42.912613+0000","last_unstale":"2023-11-08T08:18:44.272924+0000","last_undegraded":"2023-11-08T08:18:44.272924+0000","last_fullsized":"2023-11-08T08:18:44.272924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272694+0000","last_change":"2023-11-08T08:18:42.898517+0000","last_active":"2023-11-08T08:18:44.272694+0000","last_peered":"2023-11-08T08:18:44.272694+0000","last_clean":"2023-11-08T08:18:44.272694+0000","last_became_active":"2023-11-08T08:18:42.898253+0000","last_became_peered":"2023-11-08T08:18:42.898253+0000","last_unstale":"2023-11-08T08:18:44.272694+0000","last_undegraded":"2023-11-08T08:18:44.272694+0000","last_fullsized":"2023-11-08T08:18:44.272694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:56:57.876135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898142+0000","last_change":"2023-11-08T08:18:42.501016+0000","last_active":"2023-11-08T08:18:43.898142+0000","last_peered":"2023-11-08T08:18:43.898142+0000","last_clean":"2023-11-08T08:18:43.898142+0000","last_became_active":"2023-11-08T08:18:42.499597+0000","last_became_peered":"2023-11-08T08:18:42.499597+0000","last_unstale":"2023-11-08T08:18:43.898142+0000","last_undegraded":"2023-11-08T08:18:43.898142+0000","last_fullsized":"2023-11-08T08:18:43.898142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272484+0000","last_change":"2023-11-08T08:18:42.907418+0000","last_active":"2023-11-08T08:18:44.272484+0000","last_peered":"2023-11-08T08:18:44.272484+0000","last_clean":"2023-11-08T08:18:44.272484+0000","last_became_active":"2023-11-08T08:18:42.907045+0000","last_became_peered":"2023-11-08T08:18:42.907045+0000","last_unstale":"2023-11-08T08:18:44.272484+0000","last_undegraded":"2023-11-08T08:18:44.272484+0000","last_fullsized":"2023-11-08T08:18:44.272484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272507+0000","last_change":"2023-11-08T08:18:42.497426+0000","last_active":"2023-11-08T08:18:44.272507+0000","last_peered":"2023-11-08T08:18:44.272507+0000","last_clean":"2023-11-08T08:18:44.272507+0000","last_became_active":"2023-11-08T08:18:42.497130+0000","last_became_peered":"2023-11-08T08:18:42.497130+0000","last_unstale":"2023-11-08T08:18:44.272507+0000","last_undegraded":"2023-11-08T08:18:44.272507+0000","last_fullsized":"2023-11-08T08:18:44.272507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272906+0000","last_change":"2023-11-08T08:18:42.905727+0000","last_active":"2023-11-08T08:18:44.272906+0000","last_peered":"2023-11-08T08:18:44.272906+0000","last_clean":"2023-11-08T08:18:44.272906+0000","last_became_active":"2023-11-08T08:18:42.905467+0000","last_became_peered":"2023-11-08T08:18:42.905467+0000","last_unstale":"2023-11-08T08:18:44.272906+0000","last_undegraded":"2023-11-08T08:18:44.272906+0000","last_fullsized":"2023-11-08T08:18:44.272906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898639+0000","last_change":"2023-11-08T08:18:42.502963+0000","last_active":"2023-11-08T08:18:43.898639+0000","last_peered":"2023-11-08T08:18:43.898639+0000","last_clean":"2023-11-08T08:18:43.898639+0000","last_became_active":"2023-11-08T08:18:42.502702+0000","last_became_peered":"2023-11-08T08:18:42.502702+0000","last_unstale":"2023-11-08T08:18:43.898639+0000","last_undegraded":"2023-11-08T08:18:43.898639+0000","last_fullsized":"2023-11-08T08:18:43.898639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272590+0000","last_change":"2023-11-08T08:18:42.904909+0000","last_active":"2023-11-08T08:18:44.272590+0000","last_peered":"2023-11-08T08:18:44.272590+0000","last_clean":"2023-11-08T08:18:44.272590+0000","last_became_active":"2023-11-08T08:18:42.903891+0000","last_became_peered":"2023-11-08T08:18:42.903891+0000","last_unstale":"2023-11-08T08:18:44.272590+0000","last_undegraded":"2023-11-08T08:18:44.272590+0000","last_fullsized":"2023-11-08T08:18:44.272590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:21.735174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899300+0000","last_change":"2023-11-08T08:18:42.498275+0000","last_active":"2023-11-08T08:18:43.899300+0000","last_peered":"2023-11-08T08:18:43.899300+0000","last_clean":"2023-11-08T08:18:43.899300+0000","last_became_active":"2023-11-08T08:18:42.497975+0000","last_became_peered":"2023-11-08T08:18:42.497975+0000","last_unstale":"2023-11-08T08:18:43.899300+0000","last_undegraded":"2023-11-08T08:18:43.899300+0000","last_fullsized":"2023-11-08T08:18:43.899300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271740+0000","last_change":"2023-11-08T08:18:42.900693+0000","last_active":"2023-11-08T08:18:44.271740+0000","last_peered":"2023-11-08T08:18:44.271740+0000","last_clean":"2023-11-08T08:18:44.271740+0000","last_became_active":"2023-11-08T08:18:42.900407+0000","last_became_peered":"2023-11-08T08:18:42.900407+0000","last_unstale":"2023-11-08T08:18:44.271740+0000","last_undegraded":"2023-11-08T08:18:44.271740+0000","last_fullsized":"2023-11-08T08:18:44.271740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894013+0000","last_change":"2023-11-08T08:18:42.502815+0000","last_active":"2023-11-08T08:18:43.894013+0000","last_peered":"2023-11-08T08:18:43.894013+0000","last_clean":"2023-11-08T08:18:43.894013+0000","last_became_active":"2023-11-08T08:18:42.502329+0000","last_became_peered":"2023-11-08T08:18:42.502329+0000","last_unstale":"2023-11-08T08:18:43.894013+0000","last_undegraded":"2023-11-08T08:18:43.894013+0000","last_fullsized":"2023-11-08T08:18:43.894013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272456+0000","last_change":"2023-11-08T08:18:42.909334+0000","last_active":"2023-11-08T08:18:44.272456+0000","last_peered":"2023-11-08T08:18:44.272456+0000","last_clean":"2023-11-08T08:18:44.272456+0000","last_became_active":"2023-11-08T08:18:42.907416+0000","last_became_peered":"2023-11-08T08:18:42.907416+0000","last_unstale":"2023-11-08T08:18:44.272456+0000","last_undegraded":"2023-11-08T08:18:44.272456+0000","last_fullsized":"2023-11-08T08:18:44.272456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272167+0000","last_change":"2023-11-08T08:18:42.504744+0000","last_active":"2023-11-08T08:18:44.272167+0000","last_peered":"2023-11-08T08:18:44.272167+0000","last_clean":"2023-11-08T08:18:44.272167+0000","last_became_active":"2023-11-08T08:18:42.504438+0000","last_became_peered":"2023-11-08T08:18:42.504438+0000","last_unstale":"2023-11-08T08:18:44.272167+0000","last_undegraded":"2023-11-08T08:18:44.272167+0000","last_fullsized":"2023-11-08T08:18:44.272167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272646+0000","last_change":"2023-11-08T08:18:42.910922+0000","last_active":"2023-11-08T08:18:44.272646+0000","last_peered":"2023-11-08T08:18:44.272646+0000","last_clean":"2023-11-08T08:18:44.272646+0000","last_became_active":"2023-11-08T08:18:42.910578+0000","last_became_peered":"2023-11-08T08:18:42.910578+0000","last_unstale":"2023-11-08T08:18:44.272646+0000","last_undegraded":"2023-11-08T08:18:44.272646+0000","last_fullsized":"2023-11-08T08:18:44.272646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272486+0000","last_change":"2023-11-08T08:18:42.497349+0000","last_active":"2023-11-08T08:18:44.272486+0000","last_peered":"2023-11-08T08:18:44.272486+0000","last_clean":"2023-11-08T08:18:44.272486+0000","last_became_active":"2023-11-08T08:18:42.497038+0000","last_became_peered":"2023-11-08T08:18:42.497038+0000","last_unstale":"2023-11-08T08:18:44.272486+0000","last_undegraded":"2023-11-08T08:18:44.272486+0000","last_fullsized":"2023-11-08T08:18:44.272486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:12.090061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894075+0000","last_change":"2023-11-08T08:18:42.905780+0000","last_active":"2023-11-08T08:18:43.894075+0000","last_peered":"2023-11-08T08:18:43.894075+0000","last_clean":"2023-11-08T08:18:43.894075+0000","last_became_active":"2023-11-08T08:18:42.905487+0000","last_became_peered":"2023-11-08T08:18:42.905487+0000","last_unstale":"2023-11-08T08:18:43.894075+0000","last_undegraded":"2023-11-08T08:18:43.894075+0000","last_fullsized":"2023-11-08T08:18:43.894075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272417+0000","last_change":"2023-11-08T08:18:42.504787+0000","last_active":"2023-11-08T08:18:44.272417+0000","last_peered":"2023-11-08T08:18:44.272417+0000","last_clean":"2023-11-08T08:18:44.272417+0000","last_became_active":"2023-11-08T08:18:42.504477+0000","last_became_peered":"2023-11-08T08:18:42.504477+0000","last_unstale":"2023-11-08T08:18:44.272417+0000","last_undegraded":"2023-11-08T08:18:44.272417+0000","last_fullsized":"2023-11-08T08:18:44.272417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:38.511975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272076+0000","last_change":"2023-11-08T08:18:42.913147+0000","last_active":"2023-11-08T08:18:44.272076+0000","last_peered":"2023-11-08T08:18:44.272076+0000","last_clean":"2023-11-08T08:18:44.272076+0000","last_became_active":"2023-11-08T08:18:42.912855+0000","last_became_peered":"2023-11-08T08:18:42.912855+0000","last_unstale":"2023-11-08T08:18:44.272076+0000","last_undegraded":"2023-11-08T08:18:44.272076+0000","last_fullsized":"2023-11-08T08:18:44.272076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272506+0000","last_change":"2023-11-08T08:18:42.504414+0000","last_active":"2023-11-08T08:18:44.272506+0000","last_peered":"2023-11-08T08:18:44.272506+0000","last_clean":"2023-11-08T08:18:44.272506+0000","last_became_active":"2023-11-08T08:18:42.504141+0000","last_became_peered":"2023-11-08T08:18:42.504141+0000","last_unstale":"2023-11-08T08:18:44.272506+0000","last_undegraded":"2023-11-08T08:18:44.272506+0000","last_fullsized":"2023-11-08T08:18:44.272506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272647+0000","last_change":"2023-11-08T08:18:42.908975+0000","last_active":"2023-11-08T08:18:44.272647+0000","last_peered":"2023-11-08T08:18:44.272647+0000","last_clean":"2023-11-08T08:18:44.272647+0000","last_became_active":"2023-11-08T08:18:42.908137+0000","last_became_peered":"2023-11-08T08:18:42.908137+0000","last_unstale":"2023-11-08T08:18:44.272647+0000","last_undegraded":"2023-11-08T08:18:44.272647+0000","last_fullsized":"2023-11-08T08:18:44.272647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:35:59.915321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.495958+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.495711+0000","last_became_peered":"2023-11-08T08:18:42.495711+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:04.545064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899087+0000","last_change":"2023-11-08T08:18:42.907135+0000","last_active":"2023-11-08T08:18:43.899087+0000","last_peered":"2023-11-08T08:18:43.899087+0000","last_clean":"2023-11-08T08:18:43.899087+0000","last_became_active":"2023-11-08T08:18:42.906893+0000","last_became_peered":"2023-11-08T08:18:42.906893+0000","last_unstale":"2023-11-08T08:18:43.899087+0000","last_undegraded":"2023-11-08T08:18:43.899087+0000","last_fullsized":"2023-11-08T08:18:43.899087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898115+0000","last_change":"2023-11-08T08:18:42.505141+0000","last_active":"2023-11-08T08:18:43.898115+0000","last_peered":"2023-11-08T08:18:43.898115+0000","last_clean":"2023-11-08T08:18:43.898115+0000","last_became_active":"2023-11-08T08:18:42.500393+0000","last_became_peered":"2023-11-08T08:18:42.500393+0000","last_unstale":"2023-11-08T08:18:43.898115+0000","last_undegraded":"2023-11-08T08:18:43.898115+0000","last_fullsized":"2023-11-08T08:18:43.898115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899261+0000","last_change":"2023-11-08T08:18:42.908933+0000","last_active":"2023-11-08T08:18:43.899261+0000","last_peered":"2023-11-08T08:18:43.899261+0000","last_clean":"2023-11-08T08:18:43.899261+0000","last_became_active":"2023-11-08T08:18:42.908643+0000","last_became_peered":"2023-11-08T08:18:42.908643+0000","last_unstale":"2023-11-08T08:18:43.899261+0000","last_undegraded":"2023-11-08T08:18:43.899261+0000","last_fullsized":"2023-11-08T08:18:43.899261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272143+0000","last_change":"2023-11-08T08:18:42.506315+0000","last_active":"2023-11-08T08:18:44.272143+0000","last_peered":"2023-11-08T08:18:44.272143+0000","last_clean":"2023-11-08T08:18:44.272143+0000","last_became_active":"2023-11-08T08:18:42.506044+0000","last_became_peered":"2023-11-08T08:18:42.506044+0000","last_unstale":"2023-11-08T08:18:44.272143+0000","last_undegraded":"2023-11-08T08:18:44.272143+0000","last_fullsized":"2023-11-08T08:18:44.272143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271908+0000","last_change":"2023-11-08T08:18:42.908490+0000","last_active":"2023-11-08T08:18:44.271908+0000","last_peered":"2023-11-08T08:18:44.271908+0000","last_clean":"2023-11-08T08:18:44.271908+0000","last_became_active":"2023-11-08T08:18:42.908302+0000","last_became_peered":"2023-11-08T08:18:42.908302+0000","last_unstale":"2023-11-08T08:18:44.271908+0000","last_undegraded":"2023-11-08T08:18:44.271908+0000","last_fullsized":"2023-11-08T08:18:44.271908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893984+0000","last_change":"2023-11-08T08:18:42.504388+0000","last_active":"2023-11-08T08:18:43.893984+0000","last_peered":"2023-11-08T08:18:43.893984+0000","last_clean":"2023-11-08T08:18:43.893984+0000","last_became_active":"2023-11-08T08:18:42.504085+0000","last_became_peered":"2023-11-08T08:18:42.504085+0000","last_unstale":"2023-11-08T08:18:43.893984+0000","last_undegraded":"2023-11-08T08:18:43.893984+0000","last_fullsized":"2023-11-08T08:18:43.893984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898126+0000","last_change":"2023-11-08T08:18:42.901943+0000","last_active":"2023-11-08T08:18:43.898126+0000","last_peered":"2023-11-08T08:18:43.898126+0000","last_clean":"2023-11-08T08:18:43.898126+0000","last_became_active":"2023-11-08T08:18:42.901687+0000","last_became_peered":"2023-11-08T08:18:42.901687+0000","last_unstale":"2023-11-08T08:18:43.898126+0000","last_undegraded":"2023-11-08T08:18:43.898126+0000","last_fullsized":"2023-11-08T08:18:43.898126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272582+0000","last_change":"2023-11-08T08:18:42.502289+0000","last_active":"2023-11-08T08:18:44.272582+0000","last_peered":"2023-11-08T08:18:44.272582+0000","last_clean":"2023-11-08T08:18:44.272582+0000","last_became_active":"2023-11-08T08:18:42.499404+0000","last_became_peered":"2023-11-08T08:18:42.499404+0000","last_unstale":"2023-11-08T08:18:44.272582+0000","last_undegraded":"2023-11-08T08:18:44.272582+0000","last_fullsized":"2023-11-08T08:18:44.272582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:43.337457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893816+0000","last_change":"2023-11-08T08:18:42.905468+0000","last_active":"2023-11-08T08:18:43.893816+0000","last_peered":"2023-11-08T08:18:43.893816+0000","last_clean":"2023-11-08T08:18:43.893816+0000","last_became_active":"2023-11-08T08:18:42.905203+0000","last_became_peered":"2023-11-08T08:18:42.905203+0000","last_unstale":"2023-11-08T08:18:43.893816+0000","last_undegraded":"2023-11-08T08:18:43.893816+0000","last_fullsized":"2023-11-08T08:18:43.893816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.503204+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.501193+0000","last_became_peered":"2023-11-08T08:18:42.501193+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899051+0000","last_change":"2023-11-08T08:18:42.911979+0000","last_active":"2023-11-08T08:18:43.899051+0000","last_peered":"2023-11-08T08:18:43.899051+0000","last_clean":"2023-11-08T08:18:43.899051+0000","last_became_active":"2023-11-08T08:18:42.911070+0000","last_became_peered":"2023-11-08T08:18:42.911070+0000","last_unstale":"2023-11-08T08:18:43.899051+0000","last_undegraded":"2023-11-08T08:18:43.899051+0000","last_fullsized":"2023-11-08T08:18:43.899051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272150+0000","last_change":"2023-11-08T08:18:42.503625+0000","last_active":"2023-11-08T08:18:44.272150+0000","last_peered":"2023-11-08T08:18:44.272150+0000","last_clean":"2023-11-08T08:18:44.272150+0000","last_became_active":"2023-11-08T08:18:42.503351+0000","last_became_peered":"2023-11-08T08:18:42.503351+0000","last_unstale":"2023-11-08T08:18:44.272150+0000","last_undegraded":"2023-11-08T08:18:44.272150+0000","last_fullsized":"2023-11-08T08:18:44.272150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271896+0000","last_change":"2023-11-08T08:18:42.910859+0000","last_active":"2023-11-08T08:18:44.271896+0000","last_peered":"2023-11-08T08:18:44.271896+0000","last_clean":"2023-11-08T08:18:44.271896+0000","last_became_active":"2023-11-08T08:18:42.910598+0000","last_became_peered":"2023-11-08T08:18:42.910598+0000","last_unstale":"2023-11-08T08:18:44.271896+0000","last_undegraded":"2023-11-08T08:18:44.271896+0000","last_fullsized":"2023-11-08T08:18:44.271896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272298+0000","last_change":"2023-11-08T08:18:42.499085+0000","last_active":"2023-11-08T08:18:44.272298+0000","last_peered":"2023-11-08T08:18:44.272298+0000","last_clean":"2023-11-08T08:18:44.272298+0000","last_became_active":"2023-11-08T08:18:42.498750+0000","last_became_peered":"2023-11-08T08:18:42.498750+0000","last_unstale":"2023-11-08T08:18:44.272298+0000","last_undegraded":"2023-11-08T08:18:44.272298+0000","last_fullsized":"2023-11-08T08:18:44.272298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272549+0000","last_change":"2023-11-08T08:18:42.903923+0000","last_active":"2023-11-08T08:18:44.272549+0000","last_peered":"2023-11-08T08:18:44.272549+0000","last_clean":"2023-11-08T08:18:44.272549+0000","last_became_active":"2023-11-08T08:18:42.903518+0000","last_became_peered":"2023-11-08T08:18:42.903518+0000","last_unstale":"2023-11-08T08:18:44.272549+0000","last_undegraded":"2023-11-08T08:18:44.272549+0000","last_fullsized":"2023-11-08T08:18:44.272549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:51:16.185661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272139+0000","last_change":"2023-11-08T08:18:42.496450+0000","last_active":"2023-11-08T08:18:44.272139+0000","last_peered":"2023-11-08T08:18:44.272139+0000","last_clean":"2023-11-08T08:18:44.272139+0000","last_became_active":"2023-11-08T08:18:42.496197+0000","last_became_peered":"2023-11-08T08:18:42.496197+0000","last_unstale":"2023-11-08T08:18:44.272139+0000","last_undegraded":"2023-11-08T08:18:44.272139+0000","last_fullsized":"2023-11-08T08:18:44.272139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893897+0000","last_change":"2023-11-08T08:18:42.906952+0000","last_active":"2023-11-08T08:18:43.893897+0000","last_peered":"2023-11-08T08:18:43.893897+0000","last_clean":"2023-11-08T08:18:43.893897+0000","last_became_active":"2023-11-08T08:18:42.906699+0000","last_became_peered":"2023-11-08T08:18:42.906699+0000","last_unstale":"2023-11-08T08:18:43.893897+0000","last_undegraded":"2023-11-08T08:18:43.893897+0000","last_fullsized":"2023-11-08T08:18:43.893897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271876+0000","last_change":"2023-11-08T08:18:42.496216+0000","last_active":"2023-11-08T08:18:44.271876+0000","last_peered":"2023-11-08T08:18:44.271876+0000","last_clean":"2023-11-08T08:18:44.271876+0000","last_became_active":"2023-11-08T08:18:42.495940+0000","last_became_peered":"2023-11-08T08:18:42.495940+0000","last_unstale":"2023-11-08T08:18:44.271876+0000","last_undegraded":"2023-11-08T08:18:44.271876+0000","last_fullsized":"2023-11-08T08:18:44.271876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272347+0000","last_change":"2023-11-08T08:18:42.916881+0000","last_active":"2023-11-08T08:18:44.272347+0000","last_peered":"2023-11-08T08:18:44.272347+0000","last_clean":"2023-11-08T08:18:44.272347+0000","last_became_active":"2023-11-08T08:18:42.916593+0000","last_became_peered":"2023-11-08T08:18:42.916593+0000","last_unstale":"2023-11-08T08:18:44.272347+0000","last_undegraded":"2023-11-08T08:18:44.272347+0000","last_fullsized":"2023-11-08T08:18:44.272347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:09.598915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273120+0000","last_change":"2023-11-08T08:18:42.502915+0000","last_active":"2023-11-08T08:18:44.273120+0000","last_peered":"2023-11-08T08:18:44.273120+0000","last_clean":"2023-11-08T08:18:44.273120+0000","last_became_active":"2023-11-08T08:18:42.502675+0000","last_became_peered":"2023-11-08T08:18:42.502675+0000","last_unstale":"2023-11-08T08:18:44.273120+0000","last_undegraded":"2023-11-08T08:18:44.273120+0000","last_fullsized":"2023-11-08T08:18:44.273120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:08.593928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.891523+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:18:42.891523+0000","last_peered":"2023-11-08T08:18:42.891523+0000","last_clean":"2023-11-08T08:18:42.891523+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:18:42.891523+0000","last_undegraded":"2023-11-08T08:18:42.891523+0000","last_fullsized":"2023-11-08T08:18:42.891523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271984+0000","last_change":"2023-11-08T08:18:42.908748+0000","last_active":"2023-11-08T08:18:44.271984+0000","last_peered":"2023-11-08T08:18:44.271984+0000","last_clean":"2023-11-08T08:18:44.271984+0000","last_became_active":"2023-11-08T08:18:42.908419+0000","last_became_peered":"2023-11-08T08:18:42.908419+0000","last_unstale":"2023-11-08T08:18:44.271984+0000","last_undegraded":"2023-11-08T08:18:44.271984+0000","last_fullsized":"2023-11-08T08:18:44.271984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894809+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:18:42.894809+0000","last_peered":"2023-11-08T08:18:42.894809+0000","last_clean":"2023-11-08T08:18:42.894809+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:18:42.894809+0000","last_undegraded":"2023-11-08T08:18:42.894809+0000","last_fullsized":"2023-11-08T08:18:42.894809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:45.459969+0000","last_change":"2023-11-08T08:18:42.496201+0000","last_active":"2023-11-08T08:18:45.459969+0000","last_peered":"2023-11-08T08:18:45.459969+0000","last_clean":"2023-11-08T08:18:45.459969+0000","last_became_active":"2023-11-08T08:18:42.495914+0000","last_became_peered":"2023-11-08T08:18:42.495914+0000","last_unstale":"2023-11-08T08:18:45.459969+0000","last_undegraded":"2023-11-08T08:18:45.459969+0000","last_fullsized":"2023-11-08T08:18:45.459969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893727+0000","last_change":"2023-11-08T08:18:42.907389+0000","last_active":"2023-11-08T08:18:43.893727+0000","last_peered":"2023-11-08T08:18:43.893727+0000","last_clean":"2023-11-08T08:18:43.893727+0000","last_became_active":"2023-11-08T08:18:42.906767+0000","last_became_peered":"2023-11-08T08:18:42.906767+0000","last_unstale":"2023-11-08T08:18:43.893727+0000","last_undegraded":"2023-11-08T08:18:43.893727+0000","last_fullsized":"2023-11-08T08:18:43.893727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.890889+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:18:42.890889+0000","last_peered":"2023-11-08T08:18:42.890889+0000","last_clean":"2023-11-08T08:18:42.890889+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:18:42.890889+0000","last_undegraded":"2023-11-08T08:18:42.890889+0000","last_fullsized":"2023-11-08T08:18:42.890889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.895954+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:18:42.895954+0000","last_peered":"2023-11-08T08:18:42.895954+0000","last_clean":"2023-11-08T08:18:42.895954+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:18:42.895954+0000","last_undegraded":"2023-11-08T08:18:42.895954+0000","last_fullsized":"2023-11-08T08:18:42.895954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272359+0000","last_change":"2023-11-08T08:18:42.505906+0000","last_active":"2023-11-08T08:18:44.272359+0000","last_peered":"2023-11-08T08:18:44.272359+0000","last_clean":"2023-11-08T08:18:44.272359+0000","last_became_active":"2023-11-08T08:18:42.505688+0000","last_became_peered":"2023-11-08T08:18:42.505688+0000","last_unstale":"2023-11-08T08:18:44.272359+0000","last_undegraded":"2023-11-08T08:18:44.272359+0000","last_fullsized":"2023-11-08T08:18:44.272359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899202+0000","last_change":"2023-11-08T08:18:42.907668+0000","last_active":"2023-11-08T08:18:43.899202+0000","last_peered":"2023-11-08T08:18:43.899202+0000","last_clean":"2023-11-08T08:18:43.899202+0000","last_became_active":"2023-11-08T08:18:42.907352+0000","last_became_peered":"2023-11-08T08:18:42.907352+0000","last_unstale":"2023-11-08T08:18:43.899202+0000","last_undegraded":"2023-11-08T08:18:43.899202+0000","last_fullsized":"2023-11-08T08:18:43.899202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894118+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:18:42.894118+0000","last_peered":"2023-11-08T08:18:42.894118+0000","last_clean":"2023-11-08T08:18:42.894118+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:18:42.894118+0000","last_undegraded":"2023-11-08T08:18:42.894118+0000","last_fullsized":"2023-11-08T08:18:42.894118+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898738+0000","last_change":"2023-11-08T08:18:42.491954+0000","last_active":"2023-11-08T08:18:43.898738+0000","last_peered":"2023-11-08T08:18:43.898738+0000","last_clean":"2023-11-08T08:18:43.898738+0000","last_became_active":"2023-11-08T08:18:42.491305+0000","last_became_peered":"2023-11-08T08:18:42.491305+0000","last_unstale":"2023-11-08T08:18:43.898738+0000","last_undegraded":"2023-11-08T08:18:43.898738+0000","last_fullsized":"2023-11-08T08:18:43.898738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272932+0000","last_change":"2023-11-08T08:18:42.910300+0000","last_active":"2023-11-08T08:18:44.272932+0000","last_peered":"2023-11-08T08:18:44.272932+0000","last_clean":"2023-11-08T08:18:44.272932+0000","last_became_active":"2023-11-08T08:18:42.910084+0000","last_became_peered":"2023-11-08T08:18:42.910084+0000","last_unstale":"2023-11-08T08:18:44.272932+0000","last_undegraded":"2023-11-08T08:18:44.272932+0000","last_fullsized":"2023-11-08T08:18:44.272932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.889516+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:18:42.889516+0000","last_peered":"2023-11-08T08:18:42.889516+0000","last_clean":"2023-11-08T08:18:42.889516+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:18:42.889516+0000","last_undegraded":"2023-11-08T08:18:42.889516+0000","last_fullsized":"2023-11-08T08:18:42.889516+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272717+0000","last_change":"2023-11-08T08:18:42.493859+0000","last_active":"2023-11-08T08:18:44.272717+0000","last_peered":"2023-11-08T08:18:44.272717+0000","last_clean":"2023-11-08T08:18:44.272717+0000","last_became_active":"2023-11-08T08:18:42.492594+0000","last_became_peered":"2023-11-08T08:18:42.492594+0000","last_unstale":"2023-11-08T08:18:44.272717+0000","last_undegraded":"2023-11-08T08:18:44.272717+0000","last_fullsized":"2023-11-08T08:18:44.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272644+0000","last_change":"2023-11-08T08:18:42.903762+0000","last_active":"2023-11-08T08:18:44.272644+0000","last_peered":"2023-11-08T08:18:44.272644+0000","last_clean":"2023-11-08T08:18:44.272644+0000","last_became_active":"2023-11-08T08:18:42.903182+0000","last_became_peered":"2023-11-08T08:18:42.903182+0000","last_unstale":"2023-11-08T08:18:44.272644+0000","last_undegraded":"2023-11-08T08:18:44.272644+0000","last_fullsized":"2023-11-08T08:18:44.272644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:23:54.970595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896906+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:18:42.896906+0000","last_peered":"2023-11-08T08:18:42.896906+0000","last_clean":"2023-11-08T08:18:42.896906+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:18:42.896906+0000","last_undegraded":"2023-11-08T08:18:42.896906+0000","last_fullsized":"2023-11-08T08:18:42.896906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272296+0000","last_change":"2023-11-08T08:18:42.506245+0000","last_active":"2023-11-08T08:18:44.272296+0000","last_peered":"2023-11-08T08:18:44.272296+0000","last_clean":"2023-11-08T08:18:44.272296+0000","last_became_active":"2023-11-08T08:18:42.505955+0000","last_became_peered":"2023-11-08T08:18:42.505955+0000","last_unstale":"2023-11-08T08:18:44.272296+0000","last_undegraded":"2023-11-08T08:18:44.272296+0000","last_fullsized":"2023-11-08T08:18:44.272296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272936+0000","last_change":"2023-11-08T08:18:42.905213+0000","last_active":"2023-11-08T08:18:44.272936+0000","last_peered":"2023-11-08T08:18:44.272936+0000","last_clean":"2023-11-08T08:18:44.272936+0000","last_became_active":"2023-11-08T08:18:42.904949+0000","last_became_peered":"2023-11-08T08:18:42.904949+0000","last_unstale":"2023-11-08T08:18:44.272936+0000","last_undegraded":"2023-11-08T08:18:44.272936+0000","last_fullsized":"2023-11-08T08:18:44.272936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896829+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:18:42.896829+0000","last_peered":"2023-11-08T08:18:42.896829+0000","last_clean":"2023-11-08T08:18:42.896829+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:18:42.896829+0000","last_undegraded":"2023-11-08T08:18:42.896829+0000","last_fullsized":"2023-11-08T08:18:42.896829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272802+0000","last_change":"2023-11-08T08:18:42.502871+0000","last_active":"2023-11-08T08:18:44.272802+0000","last_peered":"2023-11-08T08:18:44.272802+0000","last_clean":"2023-11-08T08:18:44.272802+0000","last_became_active":"2023-11-08T08:18:42.502339+0000","last_became_peered":"2023-11-08T08:18:42.502339+0000","last_unstale":"2023-11-08T08:18:44.272802+0000","last_undegraded":"2023-11-08T08:18:44.272802+0000","last_fullsized":"2023-11-08T08:18:44.272802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:11:07.923861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272804+0000","last_change":"2023-11-08T08:18:42.910283+0000","last_active":"2023-11-08T08:18:44.272804+0000","last_peered":"2023-11-08T08:18:44.272804+0000","last_clean":"2023-11-08T08:18:44.272804+0000","last_became_active":"2023-11-08T08:18:42.909986+0000","last_became_peered":"2023-11-08T08:18:42.909986+0000","last_unstale":"2023-11-08T08:18:44.272804+0000","last_undegraded":"2023-11-08T08:18:44.272804+0000","last_fullsized":"2023-11-08T08:18:44.272804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.892038+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:18:42.892038+0000","last_peered":"2023-11-08T08:18:42.892038+0000","last_clean":"2023-11-08T08:18:42.892038+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:18:42.892038+0000","last_undegraded":"2023-11-08T08:18:42.892038+0000","last_fullsized":"2023-11-08T08:18:42.892038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893889+0000","last_change":"2023-11-08T08:18:42.503455+0000","last_active":"2023-11-08T08:18:43.893889+0000","last_peered":"2023-11-08T08:18:43.893889+0000","last_clean":"2023-11-08T08:18:43.893889+0000","last_became_active":"2023-11-08T08:18:42.503187+0000","last_became_peered":"2023-11-08T08:18:42.503187+0000","last_unstale":"2023-11-08T08:18:43.893889+0000","last_undegraded":"2023-11-08T08:18:43.893889+0000","last_fullsized":"2023-11-08T08:18:43.893889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272626+0000","last_change":"2023-11-08T08:18:42.911640+0000","last_active":"2023-11-08T08:18:44.272626+0000","last_peered":"2023-11-08T08:18:44.272626+0000","last_clean":"2023-11-08T08:18:44.272626+0000","last_became_active":"2023-11-08T08:18:42.911415+0000","last_became_peered":"2023-11-08T08:18:42.911415+0000","last_unstale":"2023-11-08T08:18:44.272626+0000","last_undegraded":"2023-11-08T08:18:44.272626+0000","last_fullsized":"2023-11-08T08:18:44.272626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272344+0000","last_change":"2023-11-08T08:18:42.501133+0000","last_active":"2023-11-08T08:18:44.272344+0000","last_peered":"2023-11-08T08:18:44.272344+0000","last_clean":"2023-11-08T08:18:44.272344+0000","last_became_active":"2023-11-08T08:18:42.500842+0000","last_became_peered":"2023-11-08T08:18:42.500842+0000","last_unstale":"2023-11-08T08:18:44.272344+0000","last_undegraded":"2023-11-08T08:18:44.272344+0000","last_fullsized":"2023-11-08T08:18:44.272344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:05.271102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272942+0000","last_change":"2023-11-08T08:18:42.905081+0000","last_active":"2023-11-08T08:18:44.272942+0000","last_peered":"2023-11-08T08:18:44.272942+0000","last_clean":"2023-11-08T08:18:44.272942+0000","last_became_active":"2023-11-08T08:18:42.904781+0000","last_became_peered":"2023-11-08T08:18:42.904781+0000","last_unstale":"2023-11-08T08:18:44.272942+0000","last_undegraded":"2023-11-08T08:18:44.272942+0000","last_fullsized":"2023-11-08T08:18:44.272942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898380+0000","last_change":"2023-11-08T08:18:42.505921+0000","last_active":"2023-11-08T08:18:43.898380+0000","last_peered":"2023-11-08T08:18:43.898380+0000","last_clean":"2023-11-08T08:18:43.898380+0000","last_became_active":"2023-11-08T08:18:42.505591+0000","last_became_peered":"2023-11-08T08:18:42.505591+0000","last_unstale":"2023-11-08T08:18:43.898380+0000","last_undegraded":"2023-11-08T08:18:43.898380+0000","last_fullsized":"2023-11-08T08:18:43.898380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893978+0000","last_change":"2023-11-08T08:18:42.907871+0000","last_active":"2023-11-08T08:18:43.893978+0000","last_peered":"2023-11-08T08:18:43.893978+0000","last_clean":"2023-11-08T08:18:43.893978+0000","last_became_active":"2023-11-08T08:18:42.907170+0000","last_became_peered":"2023-11-08T08:18:42.907170+0000","last_unstale":"2023-11-08T08:18:43.893978+0000","last_undegraded":"2023-11-08T08:18:43.893978+0000","last_fullsized":"2023-11-08T08:18:43.893978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272471+0000","last_change":"2023-11-08T08:18:42.504796+0000","last_active":"2023-11-08T08:18:44.272471+0000","last_peered":"2023-11-08T08:18:44.272471+0000","last_clean":"2023-11-08T08:18:44.272471+0000","last_became_active":"2023-11-08T08:18:42.504498+0000","last_became_peered":"2023-11-08T08:18:42.504498+0000","last_unstale":"2023-11-08T08:18:44.272471+0000","last_undegraded":"2023-11-08T08:18:44.272471+0000","last_fullsized":"2023-11-08T08:18:44.272471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:02.270931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893958+0000","last_change":"2023-11-08T08:18:42.906660+0000","last_active":"2023-11-08T08:18:43.893958+0000","last_peered":"2023-11-08T08:18:43.893958+0000","last_clean":"2023-11-08T08:18:43.893958+0000","last_became_active":"2023-11-08T08:18:42.906410+0000","last_became_peered":"2023-11-08T08:18:42.906410+0000","last_unstale":"2023-11-08T08:18:43.893958+0000","last_undegraded":"2023-11-08T08:18:43.893958+0000","last_fullsized":"2023-11-08T08:18:43.893958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899130+0000","last_change":"2023-11-08T08:18:42.497356+0000","last_active":"2023-11-08T08:18:43.899130+0000","last_peered":"2023-11-08T08:18:43.899130+0000","last_clean":"2023-11-08T08:18:43.899130+0000","last_became_active":"2023-11-08T08:18:42.497056+0000","last_became_peered":"2023-11-08T08:18:42.497056+0000","last_unstale":"2023-11-08T08:18:43.899130+0000","last_undegraded":"2023-11-08T08:18:43.899130+0000","last_fullsized":"2023-11-08T08:18:43.899130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272096+0000","last_change":"2023-11-08T08:18:42.905346+0000","last_active":"2023-11-08T08:18:44.272096+0000","last_peered":"2023-11-08T08:18:44.272096+0000","last_clean":"2023-11-08T08:18:44.272096+0000","last_became_active":"2023-11-08T08:18:42.905050+0000","last_became_peered":"2023-11-08T08:18:42.905050+0000","last_unstale":"2023-11-08T08:18:44.272096+0000","last_undegraded":"2023-11-08T08:18:44.272096+0000","last_fullsized":"2023-11-08T08:18:44.272096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899328+0000","last_change":"2023-11-08T08:18:42.499349+0000","last_active":"2023-11-08T08:18:43.899328+0000","last_peered":"2023-11-08T08:18:43.899328+0000","last_clean":"2023-11-08T08:18:43.899328+0000","last_became_active":"2023-11-08T08:18:42.499050+0000","last_became_peered":"2023-11-08T08:18:42.499050+0000","last_unstale":"2023-11-08T08:18:43.899328+0000","last_undegraded":"2023-11-08T08:18:43.899328+0000","last_fullsized":"2023-11-08T08:18:43.899328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893668+0000","last_change":"2023-11-08T08:18:42.913391+0000","last_active":"2023-11-08T08:18:43.893668+0000","last_peered":"2023-11-08T08:18:43.893668+0000","last_clean":"2023-11-08T08:18:43.893668+0000","last_became_active":"2023-11-08T08:18:42.913124+0000","last_became_peered":"2023-11-08T08:18:42.913124+0000","last_unstale":"2023-11-08T08:18:43.893668+0000","last_undegraded":"2023-11-08T08:18:43.893668+0000","last_fullsized":"2023-11-08T08:18:43.893668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898165+0000","last_change":"2023-11-08T08:18:42.912208+0000","last_active":"2023-11-08T08:18:43.898165+0000","last_peered":"2023-11-08T08:18:43.898165+0000","last_clean":"2023-11-08T08:18:43.898165+0000","last_became_active":"2023-11-08T08:18:42.911928+0000","last_became_peered":"2023-11-08T08:18:42.911928+0000","last_unstale":"2023-11-08T08:18:43.898165+0000","last_undegraded":"2023-11-08T08:18:43.898165+0000","last_fullsized":"2023-11-08T08:18:43.898165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272255+0000","last_change":"2023-11-08T08:18:42.914191+0000","last_active":"2023-11-08T08:18:44.272255+0000","last_peered":"2023-11-08T08:18:44.272255+0000","last_clean":"2023-11-08T08:18:44.272255+0000","last_became_active":"2023-11-08T08:18:42.913919+0000","last_became_peered":"2023-11-08T08:18:42.913919+0000","last_unstale":"2023-11-08T08:18:44.272255+0000","last_undegraded":"2023-11-08T08:18:44.272255+0000","last_fullsized":"2023-11-08T08:18:44.272255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271708+0000","last_change":"2023-11-08T08:18:42.900802+0000","last_active":"2023-11-08T08:18:44.271708+0000","last_peered":"2023-11-08T08:18:44.271708+0000","last_clean":"2023-11-08T08:18:44.271708+0000","last_became_active":"2023-11-08T08:18:42.900538+0000","last_became_peered":"2023-11-08T08:18:42.900538+0000","last_unstale":"2023-11-08T08:18:44.271708+0000","last_undegraded":"2023-11-08T08:18:44.271708+0000","last_fullsized":"2023-11-08T08:18:44.271708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899133+0000","last_change":"2023-11-08T08:18:42.906606+0000","last_active":"2023-11-08T08:18:43.899133+0000","last_peered":"2023-11-08T08:18:43.899133+0000","last_clean":"2023-11-08T08:18:43.899133+0000","last_became_active":"2023-11-08T08:18:42.906279+0000","last_became_peered":"2023-11-08T08:18:42.906279+0000","last_unstale":"2023-11-08T08:18:43.899133+0000","last_undegraded":"2023-11-08T08:18:43.899133+0000","last_fullsized":"2023-11-08T08:18:43.899133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899035+0000","last_change":"2023-11-08T08:18:42.915689+0000","last_active":"2023-11-08T08:18:43.899035+0000","last_peered":"2023-11-08T08:18:43.899035+0000","last_clean":"2023-11-08T08:18:43.899035+0000","last_became_active":"2023-11-08T08:18:42.915121+0000","last_became_peered":"2023-11-08T08:18:42.915121+0000","last_unstale":"2023-11-08T08:18:43.899035+0000","last_undegraded":"2023-11-08T08:18:43.899035+0000","last_fullsized":"2023-11-08T08:18:43.899035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7888,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363952,"statfs":{"total":96636764160,"available":96628686848,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":73265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":74582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9064,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362776,"statfs":{"total":96636764160,"available":96627482624,"internally_reserved":0,"allocated":819200,"data_stored":666456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9048,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362792,"statfs":{"total":96636764160,"available":96627499008,"internally_reserved":0,"allocated":802816,"data_stored":665370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:18:51.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:18:51.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.733+0000 7faee7687700 1 Processor -- start 2023-11-08T08:18:51.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.733+0000 7faee7687700 1 -- start start 2023-11-08T08:18:51.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee7687700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faee00ad900 0x7faee0072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee7687700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee00726c0 con 0x7faee006b7b0 2023-11-08T08:18:51.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee7687700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee00b43f0 con 0x7faee00b0880 2023-11-08T08:18:51.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee7687700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee006bc50 con 0x7faee00ad900 2023-11-08T08:18:51.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee5c24700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faee00ad900 0x7faee0072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee4c22700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faee006b7b0 0x7faee00ad1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34386/0 (socket says 172.21.15.116:34386) 2023-11-08T08:18:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee4c22700 1 -- 172.21.15.116:0/2081132027 learned_addr learned my addr 172.21.15.116:0/2081132027 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee5c24700 1 -- 172.21.15.116:0/2081132027 >> v1:172.21.15.129:6789/0 conn(0x7faee00b0880 legacy=0x7faee00aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee5c24700 1 -- 172.21.15.116:0/2081132027 >> v1:172.21.15.116:6789/0 conn(0x7faee006b7b0 legacy=0x7faee00ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee5c24700 1 -- 172.21.15.116:0/2081132027 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee00b46d0 con 0x7faee00ad900 2023-11-08T08:18:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faed7fff700 1 -- 172.21.15.116:0/2081132027 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769333037 0 0) 0x7faee00726c0 con 0x7faee006b7b0 2023-11-08T08:18:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.734+0000 7faee5c24700 1 --2- 172.21.15.116:0/2081132027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00ad900 0x7faee0072180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faed8009d90 tx=0x7faed800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c85c902f48a6bd7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faed7fff700 1 -- 172.21.15.116:0/2081132027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed800e070 con 0x7faee00ad900 2023-11-08T08:18:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faed7fff700 1 -- 172.21.15.116:0/2081132027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed800bb90 con 0x7faee00ad900 2023-11-08T08:18:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faed7fff700 1 -- 172.21.15.116:0/2081132027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8003a20 con 0x7faee00ad900 2023-11-08T08:18:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 -- 172.21.15.116:0/2081132027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00ad900 msgr2=0x7faee0072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 --2- 172.21.15.116:0/2081132027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00ad900 0x7faee0072180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faed8009d90 tx=0x7faed800b330 comp rx=0 tx=0).stop 2023-11-08T08:18:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 -- 172.21.15.116:0/2081132027 shutdown_connections 2023-11-08T08:18:51.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 --2- 172.21.15.116:0/2081132027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00ad900 0x7faee0072180 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 -- 172.21.15.116:0/2081132027 >> 172.21.15.116:0/2081132027 conn(0x7faee00b2fc0 msgr2=0x7faee0071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 -- 172.21.15.116:0/2081132027 shutdown_connections 2023-11-08T08:18:51.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.735+0000 7faee7687700 1 -- 172.21.15.116:0/2081132027 wait complete. 2023-11-08T08:18:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.736+0000 7faee7687700 1 Processor -- start 2023-11-08T08:18:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.736+0000 7faee7687700 1 -- start start 2023-11-08T08:18:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee7687700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faee006b7b0 0x7faee01f1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee7687700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00ad900 0x7faee00c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee7687700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 0x7faee01f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee7687700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee01f5110 con 0x7faee00b0880 2023-11-08T08:18:51.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee7687700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee01f2cd0 con 0x7faee00ad900 2023-11-08T08:18:51.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee7687700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee01f2430 con 0x7faee006b7b0 2023-11-08T08:18:51.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5c24700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 0x7faee01f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5c24700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 0x7faee01f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39454/0 (socket says 172.21.15.116:39454) 2023-11-08T08:18:51.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5c24700 1 -- 172.21.15.116:0/549834508 learned_addr learned my addr 172.21.15.116:0/549834508 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:51.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5c24700 1 -- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faee006b7b0 msgr2=0x7faee01f1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5423700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faee006b7b0 0x7faee01f1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5c24700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faee006b7b0 0x7faee01f1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee5c24700 1 -- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00ad900 msgr2=0x7faee00c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.737+0000 7faee4c22700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00ad900 0x7faee00c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faee5c24700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00ad900 0x7faee00c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faee5c24700 1 -- 172.21.15.116:0/549834508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed8009a70 con 0x7faee00b0880 2023-11-08T08:18:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faee4c22700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00ad900 0x7faee00c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faee5c24700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 0x7faee01f4bd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faee0073cc0 tx=0x7faed8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8012630 con 0x7faee00b0880 2023-11-08T08:18:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8012790 con 0x7faee00b0880 2023-11-08T08:18:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8012b40 con 0x7faee00b0880 2023-11-08T08:18:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee00b4fe0 con 0x7faee00b0880 2023-11-08T08:18:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.738+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee00b4d00 con 0x7faee00b0880 2023-11-08T08:18:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.739+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee00b49b0 con 0x7faee00b0880 2023-11-08T08:18:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.739+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faed802c070 con 0x7faee00b0880 2023-11-08T08:18:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.740+0000 7faed5ffb700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faec40682f0 0x7faec406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.740+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7faed8087ea0 con 0x7faee00b0880 2023-11-08T08:18:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.740+0000 7faee5423700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faec40682f0 0x7faec406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.740+0000 7faee5423700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faec40682f0 0x7faec406a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faee00d0030 tx=0x7faed0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.744+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faed8054700 con 0x7faee00b0880 2023-11-08T08:18:51.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.921+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faee00743f0 con 0x7faec40682f0 2023-11-08T08:18:51.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.928+0000 7faed5ffb700 1 -- 172.21.15.116:0/549834508 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286821 (secure 0 0 0) 0x7faee00743f0 con 0x7faec40682f0 2023-11-08T08:18:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:51.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.931+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faec40682f0 msgr2=0x7faec406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.931+0000 7faee7687700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faec40682f0 0x7faec406a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faee00d0030 tx=0x7faed0009040 comp rx=0 tx=0).stop 2023-11-08T08:18:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.931+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 msgr2=0x7faee01f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.931+0000 7faee7687700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 0x7faee01f4bd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faee0073cc0 tx=0x7faed8004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 shutdown_connections 2023-11-08T08:18:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee00b0880 0x7faee01f4bd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faee006b7b0 0x7faee01f1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faec40682f0 0x7faec406a7a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 --2- 172.21.15.116:0/549834508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faee00ad900 0x7faee00c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:51.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 >> 172.21.15.116:0/549834508 conn(0x7faee00b2fc0 msgr2=0x7faee0072040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:51.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 shutdown_connections 2023-11-08T08:18:51.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:51.932+0000 7faee7687700 1 -- 172.21.15.116:0/549834508 wait complete. 2023-11-08T08:18:51.937 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:18:51.943 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-11-08T08:18:50.353383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65796,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":754908924,"statfs":{"total":773094113280,"available":773026738176,"internally_reserved":0,"allocated":2887680,"data_stored":1779623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.867573"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899181+0000","last_change":"2023-11-08T08:18:42.907639+0000","last_active":"2023-11-08T08:18:43.899181+0000","last_peered":"2023-11-08T08:18:43.899181+0000","last_clean":"2023-11-08T08:18:43.899181+0000","last_became_active":"2023-11-08T08:18:42.906449+0000","last_became_peered":"2023-11-08T08:18:42.906449+0000","last_unstale":"2023-11-08T08:18:43.899181+0000","last_undegraded":"2023-11-08T08:18:43.899181+0000","last_fullsized":"2023-11-08T08:18:43.899181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893783+0000","last_change":"2023-11-08T08:18:42.912729+0000","last_active":"2023-11-08T08:18:43.893783+0000","last_peered":"2023-11-08T08:18:43.893783+0000","last_clean":"2023-11-08T08:18:43.893783+0000","last_became_active":"2023-11-08T08:18:42.912482+0000","last_became_peered":"2023-11-08T08:18:42.912482+0000","last_unstale":"2023-11-08T08:18:43.893783+0000","last_undegraded":"2023-11-08T08:18:43.893783+0000","last_fullsized":"2023-11-08T08:18:43.893783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272081+0000","last_change":"2023-11-08T08:18:42.904767+0000","last_active":"2023-11-08T08:18:44.272081+0000","last_peered":"2023-11-08T08:18:44.272081+0000","last_clean":"2023-11-08T08:18:44.272081+0000","last_became_active":"2023-11-08T08:18:42.904512+0000","last_became_peered":"2023-11-08T08:18:42.904512+0000","last_unstale":"2023-11-08T08:18:44.272081+0000","last_undegraded":"2023-11-08T08:18:44.272081+0000","last_fullsized":"2023-11-08T08:18:44.272081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271750+0000","last_change":"2023-11-08T08:18:42.901716+0000","last_active":"2023-11-08T08:18:44.271750+0000","last_peered":"2023-11-08T08:18:44.271750+0000","last_clean":"2023-11-08T08:18:44.271750+0000","last_became_active":"2023-11-08T08:18:42.901477+0000","last_became_peered":"2023-11-08T08:18:42.901477+0000","last_unstale":"2023-11-08T08:18:44.271750+0000","last_undegraded":"2023-11-08T08:18:44.271750+0000","last_fullsized":"2023-11-08T08:18:44.271750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272162+0000","last_change":"2023-11-08T08:18:42.906339+0000","last_active":"2023-11-08T08:18:44.272162+0000","last_peered":"2023-11-08T08:18:44.272162+0000","last_clean":"2023-11-08T08:18:44.272162+0000","last_became_active":"2023-11-08T08:18:42.906082+0000","last_became_peered":"2023-11-08T08:18:42.906082+0000","last_unstale":"2023-11-08T08:18:44.272162+0000","last_undegraded":"2023-11-08T08:18:44.272162+0000","last_fullsized":"2023-11-08T08:18:44.272162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272523+0000","last_change":"2023-11-08T08:18:42.907735+0000","last_active":"2023-11-08T08:18:44.272523+0000","last_peered":"2023-11-08T08:18:44.272523+0000","last_clean":"2023-11-08T08:18:44.272523+0000","last_became_active":"2023-11-08T08:18:42.907414+0000","last_became_peered":"2023-11-08T08:18:42.907414+0000","last_unstale":"2023-11-08T08:18:44.272523+0000","last_undegraded":"2023-11-08T08:18:44.272523+0000","last_fullsized":"2023-11-08T08:18:44.272523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271898+0000","last_change":"2023-11-08T08:18:42.904968+0000","last_active":"2023-11-08T08:18:44.271898+0000","last_peered":"2023-11-08T08:18:44.271898+0000","last_clean":"2023-11-08T08:18:44.271898+0000","last_became_active":"2023-11-08T08:18:42.904586+0000","last_became_peered":"2023-11-08T08:18:42.904586+0000","last_unstale":"2023-11-08T08:18:44.271898+0000","last_undegraded":"2023-11-08T08:18:44.271898+0000","last_fullsized":"2023-11-08T08:18:44.271898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:08.733824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898246+0000","last_change":"2023-11-08T08:18:42.898509+0000","last_active":"2023-11-08T08:18:43.898246+0000","last_peered":"2023-11-08T08:18:43.898246+0000","last_clean":"2023-11-08T08:18:43.898246+0000","last_became_active":"2023-11-08T08:18:42.898230+0000","last_became_peered":"2023-11-08T08:18:42.898230+0000","last_unstale":"2023-11-08T08:18:43.898246+0000","last_undegraded":"2023-11-08T08:18:43.898246+0000","last_fullsized":"2023-11-08T08:18:43.898246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273075+0000","last_change":"2023-11-08T08:18:42.906809+0000","last_active":"2023-11-08T08:18:44.273075+0000","last_peered":"2023-11-08T08:18:44.273075+0000","last_clean":"2023-11-08T08:18:44.273075+0000","last_became_active":"2023-11-08T08:18:42.906517+0000","last_became_peered":"2023-11-08T08:18:42.906517+0000","last_unstale":"2023-11-08T08:18:44.273075+0000","last_undegraded":"2023-11-08T08:18:44.273075+0000","last_fullsized":"2023-11-08T08:18:44.273075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898585+0000","last_change":"2023-11-08T08:18:42.906651+0000","last_active":"2023-11-08T08:18:43.898585+0000","last_peered":"2023-11-08T08:18:43.898585+0000","last_clean":"2023-11-08T08:18:43.898585+0000","last_became_active":"2023-11-08T08:18:42.906379+0000","last_became_peered":"2023-11-08T08:18:42.906379+0000","last_unstale":"2023-11-08T08:18:43.898585+0000","last_undegraded":"2023-11-08T08:18:43.898585+0000","last_fullsized":"2023-11-08T08:18:43.898585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898449+0000","last_change":"2023-11-08T08:18:42.898832+0000","last_active":"2023-11-08T08:18:43.898449+0000","last_peered":"2023-11-08T08:18:43.898449+0000","last_clean":"2023-11-08T08:18:43.898449+0000","last_became_active":"2023-11-08T08:18:42.898581+0000","last_became_peered":"2023-11-08T08:18:42.898581+0000","last_unstale":"2023-11-08T08:18:43.898449+0000","last_undegraded":"2023-11-08T08:18:43.898449+0000","last_fullsized":"2023-11-08T08:18:43.898449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898948+0000","last_change":"2023-11-08T08:18:42.911043+0000","last_active":"2023-11-08T08:18:43.898948+0000","last_peered":"2023-11-08T08:18:43.898948+0000","last_clean":"2023-11-08T08:18:43.898948+0000","last_became_active":"2023-11-08T08:18:42.910756+0000","last_became_peered":"2023-11-08T08:18:42.910756+0000","last_unstale":"2023-11-08T08:18:43.898948+0000","last_undegraded":"2023-11-08T08:18:43.898948+0000","last_fullsized":"2023-11-08T08:18:43.898948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272630+0000","last_change":"2023-11-08T08:18:42.915118+0000","last_active":"2023-11-08T08:18:44.272630+0000","last_peered":"2023-11-08T08:18:44.272630+0000","last_clean":"2023-11-08T08:18:44.272630+0000","last_became_active":"2023-11-08T08:18:42.914854+0000","last_became_peered":"2023-11-08T08:18:42.914854+0000","last_unstale":"2023-11-08T08:18:44.272630+0000","last_undegraded":"2023-11-08T08:18:44.272630+0000","last_fullsized":"2023-11-08T08:18:44.272630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898604+0000","last_change":"2023-11-08T08:18:42.900369+0000","last_active":"2023-11-08T08:18:43.898604+0000","last_peered":"2023-11-08T08:18:43.898604+0000","last_clean":"2023-11-08T08:18:43.898604+0000","last_became_active":"2023-11-08T08:18:42.900121+0000","last_became_peered":"2023-11-08T08:18:42.900121+0000","last_unstale":"2023-11-08T08:18:43.898604+0000","last_undegraded":"2023-11-08T08:18:43.898604+0000","last_fullsized":"2023-11-08T08:18:43.898604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271716+0000","last_change":"2023-11-08T08:18:42.912567+0000","last_active":"2023-11-08T08:18:44.271716+0000","last_peered":"2023-11-08T08:18:44.271716+0000","last_clean":"2023-11-08T08:18:44.271716+0000","last_became_active":"2023-11-08T08:18:42.912299+0000","last_became_peered":"2023-11-08T08:18:42.912299+0000","last_unstale":"2023-11-08T08:18:44.271716+0000","last_undegraded":"2023-11-08T08:18:44.271716+0000","last_fullsized":"2023-11-08T08:18:44.271716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271758+0000","last_change":"2023-11-08T08:18:42.902375+0000","last_active":"2023-11-08T08:18:44.271758+0000","last_peered":"2023-11-08T08:18:44.271758+0000","last_clean":"2023-11-08T08:18:44.271758+0000","last_became_active":"2023-11-08T08:18:42.901640+0000","last_became_peered":"2023-11-08T08:18:42.901640+0000","last_unstale":"2023-11-08T08:18:44.271758+0000","last_undegraded":"2023-11-08T08:18:44.271758+0000","last_fullsized":"2023-11-08T08:18:44.271758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272211+0000","last_change":"2023-11-08T08:18:42.905755+0000","last_active":"2023-11-08T08:18:44.272211+0000","last_peered":"2023-11-08T08:18:44.272211+0000","last_clean":"2023-11-08T08:18:44.272211+0000","last_became_active":"2023-11-08T08:18:42.905498+0000","last_became_peered":"2023-11-08T08:18:42.905498+0000","last_unstale":"2023-11-08T08:18:44.272211+0000","last_undegraded":"2023-11-08T08:18:44.272211+0000","last_fullsized":"2023-11-08T08:18:44.272211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272018+0000","last_change":"2023-11-08T08:18:42.900992+0000","last_active":"2023-11-08T08:18:44.272018+0000","last_peered":"2023-11-08T08:18:44.272018+0000","last_clean":"2023-11-08T08:18:44.272018+0000","last_became_active":"2023-11-08T08:18:42.900747+0000","last_became_peered":"2023-11-08T08:18:42.900747+0000","last_unstale":"2023-11-08T08:18:44.272018+0000","last_undegraded":"2023-11-08T08:18:44.272018+0000","last_fullsized":"2023-11-08T08:18:44.272018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899009+0000","last_change":"2023-11-08T08:18:42.910975+0000","last_active":"2023-11-08T08:18:43.899009+0000","last_peered":"2023-11-08T08:18:43.899009+0000","last_clean":"2023-11-08T08:18:43.899009+0000","last_became_active":"2023-11-08T08:18:42.910701+0000","last_became_peered":"2023-11-08T08:18:42.910701+0000","last_unstale":"2023-11-08T08:18:43.899009+0000","last_undegraded":"2023-11-08T08:18:43.899009+0000","last_fullsized":"2023-11-08T08:18:43.899009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898283+0000","last_change":"2023-11-08T08:18:42.912881+0000","last_active":"2023-11-08T08:18:43.898283+0000","last_peered":"2023-11-08T08:18:43.898283+0000","last_clean":"2023-11-08T08:18:43.898283+0000","last_became_active":"2023-11-08T08:18:42.912660+0000","last_became_peered":"2023-11-08T08:18:42.912660+0000","last_unstale":"2023-11-08T08:18:43.898283+0000","last_undegraded":"2023-11-08T08:18:43.898283+0000","last_fullsized":"2023-11-08T08:18:43.898283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898341+0000","last_change":"2023-11-08T08:18:42.901149+0000","last_active":"2023-11-08T08:18:43.898341+0000","last_peered":"2023-11-08T08:18:43.898341+0000","last_clean":"2023-11-08T08:18:43.898341+0000","last_became_active":"2023-11-08T08:18:42.900728+0000","last_became_peered":"2023-11-08T08:18:42.900728+0000","last_unstale":"2023-11-08T08:18:43.898341+0000","last_undegraded":"2023-11-08T08:18:43.898341+0000","last_fullsized":"2023-11-08T08:18:43.898341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272020+0000","last_change":"2023-11-08T08:18:42.911764+0000","last_active":"2023-11-08T08:18:44.272020+0000","last_peered":"2023-11-08T08:18:44.272020+0000","last_clean":"2023-11-08T08:18:44.272020+0000","last_became_active":"2023-11-08T08:18:42.911477+0000","last_became_peered":"2023-11-08T08:18:42.911477+0000","last_unstale":"2023-11-08T08:18:44.272020+0000","last_undegraded":"2023-11-08T08:18:44.272020+0000","last_fullsized":"2023-11-08T08:18:44.272020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272570+0000","last_change":"2023-11-08T08:18:42.909892+0000","last_active":"2023-11-08T08:18:44.272570+0000","last_peered":"2023-11-08T08:18:44.272570+0000","last_clean":"2023-11-08T08:18:44.272570+0000","last_became_active":"2023-11-08T08:18:42.909646+0000","last_became_peered":"2023-11-08T08:18:42.909646+0000","last_unstale":"2023-11-08T08:18:44.272570+0000","last_undegraded":"2023-11-08T08:18:44.272570+0000","last_fullsized":"2023-11-08T08:18:44.272570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898218+0000","last_change":"2023-11-08T08:18:42.897278+0000","last_active":"2023-11-08T08:18:43.898218+0000","last_peered":"2023-11-08T08:18:43.898218+0000","last_clean":"2023-11-08T08:18:43.898218+0000","last_became_active":"2023-11-08T08:18:42.896923+0000","last_became_peered":"2023-11-08T08:18:42.896923+0000","last_unstale":"2023-11-08T08:18:43.898218+0000","last_undegraded":"2023-11-08T08:18:43.898218+0000","last_fullsized":"2023-11-08T08:18:43.898218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:21.034079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273144+0000","last_change":"2023-11-08T08:18:42.909514+0000","last_active":"2023-11-08T08:18:44.273144+0000","last_peered":"2023-11-08T08:18:44.273144+0000","last_clean":"2023-11-08T08:18:44.273144+0000","last_became_active":"2023-11-08T08:18:42.909246+0000","last_became_peered":"2023-11-08T08:18:42.909246+0000","last_unstale":"2023-11-08T08:18:44.273144+0000","last_undegraded":"2023-11-08T08:18:44.273144+0000","last_fullsized":"2023-11-08T08:18:44.273144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272924+0000","last_change":"2023-11-08T08:18:42.912860+0000","last_active":"2023-11-08T08:18:44.272924+0000","last_peered":"2023-11-08T08:18:44.272924+0000","last_clean":"2023-11-08T08:18:44.272924+0000","last_became_active":"2023-11-08T08:18:42.912613+0000","last_became_peered":"2023-11-08T08:18:42.912613+0000","last_unstale":"2023-11-08T08:18:44.272924+0000","last_undegraded":"2023-11-08T08:18:44.272924+0000","last_fullsized":"2023-11-08T08:18:44.272924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272694+0000","last_change":"2023-11-08T08:18:42.898517+0000","last_active":"2023-11-08T08:18:44.272694+0000","last_peered":"2023-11-08T08:18:44.272694+0000","last_clean":"2023-11-08T08:18:44.272694+0000","last_became_active":"2023-11-08T08:18:42.898253+0000","last_became_peered":"2023-11-08T08:18:42.898253+0000","last_unstale":"2023-11-08T08:18:44.272694+0000","last_undegraded":"2023-11-08T08:18:44.272694+0000","last_fullsized":"2023-11-08T08:18:44.272694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:56:57.876135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898142+0000","last_change":"2023-11-08T08:18:42.501016+0000","last_active":"2023-11-08T08:18:43.898142+0000","last_peered":"2023-11-08T08:18:43.898142+0000","last_clean":"2023-11-08T08:18:43.898142+0000","last_became_active":"2023-11-08T08:18:42.499597+0000","last_became_peered":"2023-11-08T08:18:42.499597+0000","last_unstale":"2023-11-08T08:18:43.898142+0000","last_undegraded":"2023-11-08T08:18:43.898142+0000","last_fullsized":"2023-11-08T08:18:43.898142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272484+0000","last_change":"2023-11-08T08:18:42.907418+0000","last_active":"2023-11-08T08:18:44.272484+0000","last_peered":"2023-11-08T08:18:44.272484+0000","last_clean":"2023-11-08T08:18:44.272484+0000","last_became_active":"2023-11-08T08:18:42.907045+0000","last_became_peered":"2023-11-08T08:18:42.907045+0000","last_unstale":"2023-11-08T08:18:44.272484+0000","last_undegraded":"2023-11-08T08:18:44.272484+0000","last_fullsized":"2023-11-08T08:18:44.272484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272507+0000","last_change":"2023-11-08T08:18:42.497426+0000","last_active":"2023-11-08T08:18:44.272507+0000","last_peered":"2023-11-08T08:18:44.272507+0000","last_clean":"2023-11-08T08:18:44.272507+0000","last_became_active":"2023-11-08T08:18:42.497130+0000","last_became_peered":"2023-11-08T08:18:42.497130+0000","last_unstale":"2023-11-08T08:18:44.272507+0000","last_undegraded":"2023-11-08T08:18:44.272507+0000","last_fullsized":"2023-11-08T08:18:44.272507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272906+0000","last_change":"2023-11-08T08:18:42.905727+0000","last_active":"2023-11-08T08:18:44.272906+0000","last_peered":"2023-11-08T08:18:44.272906+0000","last_clean":"2023-11-08T08:18:44.272906+0000","last_became_active":"2023-11-08T08:18:42.905467+0000","last_became_peered":"2023-11-08T08:18:42.905467+0000","last_unstale":"2023-11-08T08:18:44.272906+0000","last_undegraded":"2023-11-08T08:18:44.272906+0000","last_fullsized":"2023-11-08T08:18:44.272906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898639+0000","last_change":"2023-11-08T08:18:42.502963+0000","last_active":"2023-11-08T08:18:43.898639+0000","last_peered":"2023-11-08T08:18:43.898639+0000","last_clean":"2023-11-08T08:18:43.898639+0000","last_became_active":"2023-11-08T08:18:42.502702+0000","last_became_peered":"2023-11-08T08:18:42.502702+0000","last_unstale":"2023-11-08T08:18:43.898639+0000","last_undegraded":"2023-11-08T08:18:43.898639+0000","last_fullsized":"2023-11-08T08:18:43.898639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272590+0000","last_change":"2023-11-08T08:18:42.904909+0000","last_active":"2023-11-08T08:18:44.272590+0000","last_peered":"2023-11-08T08:18:44.272590+0000","last_clean":"2023-11-08T08:18:44.272590+0000","last_became_active":"2023-11-08T08:18:42.903891+0000","last_became_peered":"2023-11-08T08:18:42.903891+0000","last_unstale":"2023-11-08T08:18:44.272590+0000","last_undegraded":"2023-11-08T08:18:44.272590+0000","last_fullsized":"2023-11-08T08:18:44.272590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:21.735174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899300+0000","last_change":"2023-11-08T08:18:42.498275+0000","last_active":"2023-11-08T08:18:43.899300+0000","last_peered":"2023-11-08T08:18:43.899300+0000","last_clean":"2023-11-08T08:18:43.899300+0000","last_became_active":"2023-11-08T08:18:42.497975+0000","last_became_peered":"2023-11-08T08:18:42.497975+0000","last_unstale":"2023-11-08T08:18:43.899300+0000","last_undegraded":"2023-11-08T08:18:43.899300+0000","last_fullsized":"2023-11-08T08:18:43.899300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271740+0000","last_change":"2023-11-08T08:18:42.900693+0000","last_active":"2023-11-08T08:18:44.271740+0000","last_peered":"2023-11-08T08:18:44.271740+0000","last_clean":"2023-11-08T08:18:44.271740+0000","last_became_active":"2023-11-08T08:18:42.900407+0000","last_became_peered":"2023-11-08T08:18:42.900407+0000","last_unstale":"2023-11-08T08:18:44.271740+0000","last_undegraded":"2023-11-08T08:18:44.271740+0000","last_fullsized":"2023-11-08T08:18:44.271740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894013+0000","last_change":"2023-11-08T08:18:42.502815+0000","last_active":"2023-11-08T08:18:43.894013+0000","last_peered":"2023-11-08T08:18:43.894013+0000","last_clean":"2023-11-08T08:18:43.894013+0000","last_became_active":"2023-11-08T08:18:42.502329+0000","last_became_peered":"2023-11-08T08:18:42.502329+0000","last_unstale":"2023-11-08T08:18:43.894013+0000","last_undegraded":"2023-11-08T08:18:43.894013+0000","last_fullsized":"2023-11-08T08:18:43.894013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272456+0000","last_change":"2023-11-08T08:18:42.909334+0000","last_active":"2023-11-08T08:18:44.272456+0000","last_peered":"2023-11-08T08:18:44.272456+0000","last_clean":"2023-11-08T08:18:44.272456+0000","last_became_active":"2023-11-08T08:18:42.907416+0000","last_became_peered":"2023-11-08T08:18:42.907416+0000","last_unstale":"2023-11-08T08:18:44.272456+0000","last_undegraded":"2023-11-08T08:18:44.272456+0000","last_fullsized":"2023-11-08T08:18:44.272456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:13:53.848431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272167+0000","last_change":"2023-11-08T08:18:42.504744+0000","last_active":"2023-11-08T08:18:44.272167+0000","last_peered":"2023-11-08T08:18:44.272167+0000","last_clean":"2023-11-08T08:18:44.272167+0000","last_became_active":"2023-11-08T08:18:42.504438+0000","last_became_peered":"2023-11-08T08:18:42.504438+0000","last_unstale":"2023-11-08T08:18:44.272167+0000","last_undegraded":"2023-11-08T08:18:44.272167+0000","last_fullsized":"2023-11-08T08:18:44.272167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272646+0000","last_change":"2023-11-08T08:18:42.910922+0000","last_active":"2023-11-08T08:18:44.272646+0000","last_peered":"2023-11-08T08:18:44.272646+0000","last_clean":"2023-11-08T08:18:44.272646+0000","last_became_active":"2023-11-08T08:18:42.910578+0000","last_became_peered":"2023-11-08T08:18:42.910578+0000","last_unstale":"2023-11-08T08:18:44.272646+0000","last_undegraded":"2023-11-08T08:18:44.272646+0000","last_fullsized":"2023-11-08T08:18:44.272646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272486+0000","last_change":"2023-11-08T08:18:42.497349+0000","last_active":"2023-11-08T08:18:44.272486+0000","last_peered":"2023-11-08T08:18:44.272486+0000","last_clean":"2023-11-08T08:18:44.272486+0000","last_became_active":"2023-11-08T08:18:42.497038+0000","last_became_peered":"2023-11-08T08:18:42.497038+0000","last_unstale":"2023-11-08T08:18:44.272486+0000","last_undegraded":"2023-11-08T08:18:44.272486+0000","last_fullsized":"2023-11-08T08:18:44.272486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:12.090061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.894075+0000","last_change":"2023-11-08T08:18:42.905780+0000","last_active":"2023-11-08T08:18:43.894075+0000","last_peered":"2023-11-08T08:18:43.894075+0000","last_clean":"2023-11-08T08:18:43.894075+0000","last_became_active":"2023-11-08T08:18:42.905487+0000","last_became_peered":"2023-11-08T08:18:42.905487+0000","last_unstale":"2023-11-08T08:18:43.894075+0000","last_undegraded":"2023-11-08T08:18:43.894075+0000","last_fullsized":"2023-11-08T08:18:43.894075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272417+0000","last_change":"2023-11-08T08:18:42.504787+0000","last_active":"2023-11-08T08:18:44.272417+0000","last_peered":"2023-11-08T08:18:44.272417+0000","last_clean":"2023-11-08T08:18:44.272417+0000","last_became_active":"2023-11-08T08:18:42.504477+0000","last_became_peered":"2023-11-08T08:18:42.504477+0000","last_unstale":"2023-11-08T08:18:44.272417+0000","last_undegraded":"2023-11-08T08:18:44.272417+0000","last_fullsized":"2023-11-08T08:18:44.272417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:38.511975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272076+0000","last_change":"2023-11-08T08:18:42.913147+0000","last_active":"2023-11-08T08:18:44.272076+0000","last_peered":"2023-11-08T08:18:44.272076+0000","last_clean":"2023-11-08T08:18:44.272076+0000","last_became_active":"2023-11-08T08:18:42.912855+0000","last_became_peered":"2023-11-08T08:18:42.912855+0000","last_unstale":"2023-11-08T08:18:44.272076+0000","last_undegraded":"2023-11-08T08:18:44.272076+0000","last_fullsized":"2023-11-08T08:18:44.272076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:05.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272506+0000","last_change":"2023-11-08T08:18:42.504414+0000","last_active":"2023-11-08T08:18:44.272506+0000","last_peered":"2023-11-08T08:18:44.272506+0000","last_clean":"2023-11-08T08:18:44.272506+0000","last_became_active":"2023-11-08T08:18:42.504141+0000","last_became_peered":"2023-11-08T08:18:42.504141+0000","last_unstale":"2023-11-08T08:18:44.272506+0000","last_undegraded":"2023-11-08T08:18:44.272506+0000","last_fullsized":"2023-11-08T08:18:44.272506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272647+0000","last_change":"2023-11-08T08:18:42.908975+0000","last_active":"2023-11-08T08:18:44.272647+0000","last_peered":"2023-11-08T08:18:44.272647+0000","last_clean":"2023-11-08T08:18:44.272647+0000","last_became_active":"2023-11-08T08:18:42.908137+0000","last_became_peered":"2023-11-08T08:18:42.908137+0000","last_unstale":"2023-11-08T08:18:44.272647+0000","last_undegraded":"2023-11-08T08:18:44.272647+0000","last_fullsized":"2023-11-08T08:18:44.272647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:35:59.915321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.495958+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.495711+0000","last_became_peered":"2023-11-08T08:18:42.495711+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:04.545064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899087+0000","last_change":"2023-11-08T08:18:42.907135+0000","last_active":"2023-11-08T08:18:43.899087+0000","last_peered":"2023-11-08T08:18:43.899087+0000","last_clean":"2023-11-08T08:18:43.899087+0000","last_became_active":"2023-11-08T08:18:42.906893+0000","last_became_peered":"2023-11-08T08:18:42.906893+0000","last_unstale":"2023-11-08T08:18:43.899087+0000","last_undegraded":"2023-11-08T08:18:43.899087+0000","last_fullsized":"2023-11-08T08:18:43.899087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898115+0000","last_change":"2023-11-08T08:18:42.505141+0000","last_active":"2023-11-08T08:18:43.898115+0000","last_peered":"2023-11-08T08:18:43.898115+0000","last_clean":"2023-11-08T08:18:43.898115+0000","last_became_active":"2023-11-08T08:18:42.500393+0000","last_became_peered":"2023-11-08T08:18:42.500393+0000","last_unstale":"2023-11-08T08:18:43.898115+0000","last_undegraded":"2023-11-08T08:18:43.898115+0000","last_fullsized":"2023-11-08T08:18:43.898115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899261+0000","last_change":"2023-11-08T08:18:42.908933+0000","last_active":"2023-11-08T08:18:43.899261+0000","last_peered":"2023-11-08T08:18:43.899261+0000","last_clean":"2023-11-08T08:18:43.899261+0000","last_became_active":"2023-11-08T08:18:42.908643+0000","last_became_peered":"2023-11-08T08:18:42.908643+0000","last_unstale":"2023-11-08T08:18:43.899261+0000","last_undegraded":"2023-11-08T08:18:43.899261+0000","last_fullsized":"2023-11-08T08:18:43.899261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:01:54.093091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272143+0000","last_change":"2023-11-08T08:18:42.506315+0000","last_active":"2023-11-08T08:18:44.272143+0000","last_peered":"2023-11-08T08:18:44.272143+0000","last_clean":"2023-11-08T08:18:44.272143+0000","last_became_active":"2023-11-08T08:18:42.506044+0000","last_became_peered":"2023-11-08T08:18:42.506044+0000","last_unstale":"2023-11-08T08:18:44.272143+0000","last_undegraded":"2023-11-08T08:18:44.272143+0000","last_fullsized":"2023-11-08T08:18:44.272143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271908+0000","last_change":"2023-11-08T08:18:42.908490+0000","last_active":"2023-11-08T08:18:44.271908+0000","last_peered":"2023-11-08T08:18:44.271908+0000","last_clean":"2023-11-08T08:18:44.271908+0000","last_became_active":"2023-11-08T08:18:42.908302+0000","last_became_peered":"2023-11-08T08:18:42.908302+0000","last_unstale":"2023-11-08T08:18:44.271908+0000","last_undegraded":"2023-11-08T08:18:44.271908+0000","last_fullsized":"2023-11-08T08:18:44.271908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:17:45.658434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893984+0000","last_change":"2023-11-08T08:18:42.504388+0000","last_active":"2023-11-08T08:18:43.893984+0000","last_peered":"2023-11-08T08:18:43.893984+0000","last_clean":"2023-11-08T08:18:43.893984+0000","last_became_active":"2023-11-08T08:18:42.504085+0000","last_became_peered":"2023-11-08T08:18:42.504085+0000","last_unstale":"2023-11-08T08:18:43.893984+0000","last_undegraded":"2023-11-08T08:18:43.893984+0000","last_fullsized":"2023-11-08T08:18:43.893984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898126+0000","last_change":"2023-11-08T08:18:42.901943+0000","last_active":"2023-11-08T08:18:43.898126+0000","last_peered":"2023-11-08T08:18:43.898126+0000","last_clean":"2023-11-08T08:18:43.898126+0000","last_became_active":"2023-11-08T08:18:42.901687+0000","last_became_peered":"2023-11-08T08:18:42.901687+0000","last_unstale":"2023-11-08T08:18:43.898126+0000","last_undegraded":"2023-11-08T08:18:43.898126+0000","last_fullsized":"2023-11-08T08:18:43.898126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:33:30.838242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272582+0000","last_change":"2023-11-08T08:18:42.502289+0000","last_active":"2023-11-08T08:18:44.272582+0000","last_peered":"2023-11-08T08:18:44.272582+0000","last_clean":"2023-11-08T08:18:44.272582+0000","last_became_active":"2023-11-08T08:18:42.499404+0000","last_became_peered":"2023-11-08T08:18:42.499404+0000","last_unstale":"2023-11-08T08:18:44.272582+0000","last_undegraded":"2023-11-08T08:18:44.272582+0000","last_fullsized":"2023-11-08T08:18:44.272582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:43.337457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893816+0000","last_change":"2023-11-08T08:18:42.905468+0000","last_active":"2023-11-08T08:18:43.893816+0000","last_peered":"2023-11-08T08:18:43.893816+0000","last_clean":"2023-11-08T08:18:43.893816+0000","last_became_active":"2023-11-08T08:18:42.905203+0000","last_became_peered":"2023-11-08T08:18:42.905203+0000","last_unstale":"2023-11-08T08:18:43.893816+0000","last_undegraded":"2023-11-08T08:18:43.893816+0000","last_fullsized":"2023-11-08T08:18:43.893816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271979+0000","last_change":"2023-11-08T08:18:42.503204+0000","last_active":"2023-11-08T08:18:44.271979+0000","last_peered":"2023-11-08T08:18:44.271979+0000","last_clean":"2023-11-08T08:18:44.271979+0000","last_became_active":"2023-11-08T08:18:42.501193+0000","last_became_peered":"2023-11-08T08:18:42.501193+0000","last_unstale":"2023-11-08T08:18:44.271979+0000","last_undegraded":"2023-11-08T08:18:44.271979+0000","last_fullsized":"2023-11-08T08:18:44.271979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:03:25.203275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899051+0000","last_change":"2023-11-08T08:18:42.911979+0000","last_active":"2023-11-08T08:18:43.899051+0000","last_peered":"2023-11-08T08:18:43.899051+0000","last_clean":"2023-11-08T08:18:43.899051+0000","last_became_active":"2023-11-08T08:18:42.911070+0000","last_became_peered":"2023-11-08T08:18:42.911070+0000","last_unstale":"2023-11-08T08:18:43.899051+0000","last_undegraded":"2023-11-08T08:18:43.899051+0000","last_fullsized":"2023-11-08T08:18:43.899051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272150+0000","last_change":"2023-11-08T08:18:42.503625+0000","last_active":"2023-11-08T08:18:44.272150+0000","last_peered":"2023-11-08T08:18:44.272150+0000","last_clean":"2023-11-08T08:18:44.272150+0000","last_became_active":"2023-11-08T08:18:42.503351+0000","last_became_peered":"2023-11-08T08:18:42.503351+0000","last_unstale":"2023-11-08T08:18:44.272150+0000","last_undegraded":"2023-11-08T08:18:44.272150+0000","last_fullsized":"2023-11-08T08:18:44.272150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271896+0000","last_change":"2023-11-08T08:18:42.910859+0000","last_active":"2023-11-08T08:18:44.271896+0000","last_peered":"2023-11-08T08:18:44.271896+0000","last_clean":"2023-11-08T08:18:44.271896+0000","last_became_active":"2023-11-08T08:18:42.910598+0000","last_became_peered":"2023-11-08T08:18:42.910598+0000","last_unstale":"2023-11-08T08:18:44.271896+0000","last_undegraded":"2023-11-08T08:18:44.271896+0000","last_fullsized":"2023-11-08T08:18:44.271896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272298+0000","last_change":"2023-11-08T08:18:42.499085+0000","last_active":"2023-11-08T08:18:44.272298+0000","last_peered":"2023-11-08T08:18:44.272298+0000","last_clean":"2023-11-08T08:18:44.272298+0000","last_became_active":"2023-11-08T08:18:42.498750+0000","last_became_peered":"2023-11-08T08:18:42.498750+0000","last_unstale":"2023-11-08T08:18:44.272298+0000","last_undegraded":"2023-11-08T08:18:44.272298+0000","last_fullsized":"2023-11-08T08:18:44.272298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:51:44.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272549+0000","last_change":"2023-11-08T08:18:42.903923+0000","last_active":"2023-11-08T08:18:44.272549+0000","last_peered":"2023-11-08T08:18:44.272549+0000","last_clean":"2023-11-08T08:18:44.272549+0000","last_became_active":"2023-11-08T08:18:42.903518+0000","last_became_peered":"2023-11-08T08:18:42.903518+0000","last_unstale":"2023-11-08T08:18:44.272549+0000","last_undegraded":"2023-11-08T08:18:44.272549+0000","last_fullsized":"2023-11-08T08:18:44.272549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:51:16.185661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272139+0000","last_change":"2023-11-08T08:18:42.496450+0000","last_active":"2023-11-08T08:18:44.272139+0000","last_peered":"2023-11-08T08:18:44.272139+0000","last_clean":"2023-11-08T08:18:44.272139+0000","last_became_active":"2023-11-08T08:18:42.496197+0000","last_became_peered":"2023-11-08T08:18:42.496197+0000","last_unstale":"2023-11-08T08:18:44.272139+0000","last_undegraded":"2023-11-08T08:18:44.272139+0000","last_fullsized":"2023-11-08T08:18:44.272139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893897+0000","last_change":"2023-11-08T08:18:42.906952+0000","last_active":"2023-11-08T08:18:43.893897+0000","last_peered":"2023-11-08T08:18:43.893897+0000","last_clean":"2023-11-08T08:18:43.893897+0000","last_became_active":"2023-11-08T08:18:42.906699+0000","last_became_peered":"2023-11-08T08:18:42.906699+0000","last_unstale":"2023-11-08T08:18:43.893897+0000","last_undegraded":"2023-11-08T08:18:43.893897+0000","last_fullsized":"2023-11-08T08:18:43.893897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271876+0000","last_change":"2023-11-08T08:18:42.496216+0000","last_active":"2023-11-08T08:18:44.271876+0000","last_peered":"2023-11-08T08:18:44.271876+0000","last_clean":"2023-11-08T08:18:44.271876+0000","last_became_active":"2023-11-08T08:18:42.495940+0000","last_became_peered":"2023-11-08T08:18:42.495940+0000","last_unstale":"2023-11-08T08:18:44.271876+0000","last_undegraded":"2023-11-08T08:18:44.271876+0000","last_fullsized":"2023-11-08T08:18:44.271876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272347+0000","last_change":"2023-11-08T08:18:42.916881+0000","last_active":"2023-11-08T08:18:44.272347+0000","last_peered":"2023-11-08T08:18:44.272347+0000","last_clean":"2023-11-08T08:18:44.272347+0000","last_became_active":"2023-11-08T08:18:42.916593+0000","last_became_peered":"2023-11-08T08:18:42.916593+0000","last_unstale":"2023-11-08T08:18:44.272347+0000","last_undegraded":"2023-11-08T08:18:44.272347+0000","last_fullsized":"2023-11-08T08:18:44.272347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:09.598915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.273120+0000","last_change":"2023-11-08T08:18:42.502915+0000","last_active":"2023-11-08T08:18:44.273120+0000","last_peered":"2023-11-08T08:18:44.273120+0000","last_clean":"2023-11-08T08:18:44.273120+0000","last_became_active":"2023-11-08T08:18:42.502675+0000","last_became_peered":"2023-11-08T08:18:42.502675+0000","last_unstale":"2023-11-08T08:18:44.273120+0000","last_undegraded":"2023-11-08T08:18:44.273120+0000","last_fullsized":"2023-11-08T08:18:44.273120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:59:08.593928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.891523+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:18:42.891523+0000","last_peered":"2023-11-08T08:18:42.891523+0000","last_clean":"2023-11-08T08:18:42.891523+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:18:42.891523+0000","last_undegraded":"2023-11-08T08:18:42.891523+0000","last_fullsized":"2023-11-08T08:18:42.891523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271984+0000","last_change":"2023-11-08T08:18:42.908748+0000","last_active":"2023-11-08T08:18:44.271984+0000","last_peered":"2023-11-08T08:18:44.271984+0000","last_clean":"2023-11-08T08:18:44.271984+0000","last_became_active":"2023-11-08T08:18:42.908419+0000","last_became_peered":"2023-11-08T08:18:42.908419+0000","last_unstale":"2023-11-08T08:18:44.271984+0000","last_undegraded":"2023-11-08T08:18:44.271984+0000","last_fullsized":"2023-11-08T08:18:44.271984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:19.208169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894809+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:18:42.894809+0000","last_peered":"2023-11-08T08:18:42.894809+0000","last_clean":"2023-11-08T08:18:42.894809+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:18:42.894809+0000","last_undegraded":"2023-11-08T08:18:42.894809+0000","last_fullsized":"2023-11-08T08:18:42.894809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:45.459969+0000","last_change":"2023-11-08T08:18:42.496201+0000","last_active":"2023-11-08T08:18:45.459969+0000","last_peered":"2023-11-08T08:18:45.459969+0000","last_clean":"2023-11-08T08:18:45.459969+0000","last_became_active":"2023-11-08T08:18:42.495914+0000","last_became_peered":"2023-11-08T08:18:42.495914+0000","last_unstale":"2023-11-08T08:18:45.459969+0000","last_undegraded":"2023-11-08T08:18:45.459969+0000","last_fullsized":"2023-11-08T08:18:45.459969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893727+0000","last_change":"2023-11-08T08:18:42.907389+0000","last_active":"2023-11-08T08:18:43.893727+0000","last_peered":"2023-11-08T08:18:43.893727+0000","last_clean":"2023-11-08T08:18:43.893727+0000","last_became_active":"2023-11-08T08:18:42.906767+0000","last_became_peered":"2023-11-08T08:18:42.906767+0000","last_unstale":"2023-11-08T08:18:43.893727+0000","last_undegraded":"2023-11-08T08:18:43.893727+0000","last_fullsized":"2023-11-08T08:18:43.893727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:34:45.319661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.890889+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:18:42.890889+0000","last_peered":"2023-11-08T08:18:42.890889+0000","last_clean":"2023-11-08T08:18:42.890889+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:18:42.890889+0000","last_undegraded":"2023-11-08T08:18:42.890889+0000","last_fullsized":"2023-11-08T08:18:42.890889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.895954+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:18:42.895954+0000","last_peered":"2023-11-08T08:18:42.895954+0000","last_clean":"2023-11-08T08:18:42.895954+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:18:42.895954+0000","last_undegraded":"2023-11-08T08:18:42.895954+0000","last_fullsized":"2023-11-08T08:18:42.895954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272359+0000","last_change":"2023-11-08T08:18:42.505906+0000","last_active":"2023-11-08T08:18:44.272359+0000","last_peered":"2023-11-08T08:18:44.272359+0000","last_clean":"2023-11-08T08:18:44.272359+0000","last_became_active":"2023-11-08T08:18:42.505688+0000","last_became_peered":"2023-11-08T08:18:42.505688+0000","last_unstale":"2023-11-08T08:18:44.272359+0000","last_undegraded":"2023-11-08T08:18:44.272359+0000","last_fullsized":"2023-11-08T08:18:44.272359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:02:47.676078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899202+0000","last_change":"2023-11-08T08:18:42.907668+0000","last_active":"2023-11-08T08:18:43.899202+0000","last_peered":"2023-11-08T08:18:43.899202+0000","last_clean":"2023-11-08T08:18:43.899202+0000","last_became_active":"2023-11-08T08:18:42.907352+0000","last_became_peered":"2023-11-08T08:18:42.907352+0000","last_unstale":"2023-11-08T08:18:43.899202+0000","last_undegraded":"2023-11-08T08:18:43.899202+0000","last_fullsized":"2023-11-08T08:18:43.899202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:27:38.602390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.894118+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:18:42.894118+0000","last_peered":"2023-11-08T08:18:42.894118+0000","last_clean":"2023-11-08T08:18:42.894118+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:18:42.894118+0000","last_undegraded":"2023-11-08T08:18:42.894118+0000","last_fullsized":"2023-11-08T08:18:42.894118+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898738+0000","last_change":"2023-11-08T08:18:42.491954+0000","last_active":"2023-11-08T08:18:43.898738+0000","last_peered":"2023-11-08T08:18:43.898738+0000","last_clean":"2023-11-08T08:18:43.898738+0000","last_became_active":"2023-11-08T08:18:42.491305+0000","last_became_peered":"2023-11-08T08:18:42.491305+0000","last_unstale":"2023-11-08T08:18:43.898738+0000","last_undegraded":"2023-11-08T08:18:43.898738+0000","last_fullsized":"2023-11-08T08:18:43.898738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:24:50.785756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272932+0000","last_change":"2023-11-08T08:18:42.910300+0000","last_active":"2023-11-08T08:18:44.272932+0000","last_peered":"2023-11-08T08:18:44.272932+0000","last_clean":"2023-11-08T08:18:44.272932+0000","last_became_active":"2023-11-08T08:18:42.910084+0000","last_became_peered":"2023-11-08T08:18:42.910084+0000","last_unstale":"2023-11-08T08:18:44.272932+0000","last_undegraded":"2023-11-08T08:18:44.272932+0000","last_fullsized":"2023-11-08T08:18:44.272932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:03.275919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.889516+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:18:42.889516+0000","last_peered":"2023-11-08T08:18:42.889516+0000","last_clean":"2023-11-08T08:18:42.889516+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:18:42.889516+0000","last_undegraded":"2023-11-08T08:18:42.889516+0000","last_fullsized":"2023-11-08T08:18:42.889516+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272717+0000","last_change":"2023-11-08T08:18:42.493859+0000","last_active":"2023-11-08T08:18:44.272717+0000","last_peered":"2023-11-08T08:18:44.272717+0000","last_clean":"2023-11-08T08:18:44.272717+0000","last_became_active":"2023-11-08T08:18:42.492594+0000","last_became_peered":"2023-11-08T08:18:42.492594+0000","last_unstale":"2023-11-08T08:18:44.272717+0000","last_undegraded":"2023-11-08T08:18:44.272717+0000","last_fullsized":"2023-11-08T08:18:44.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:20:59.794058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272644+0000","last_change":"2023-11-08T08:18:42.903762+0000","last_active":"2023-11-08T08:18:44.272644+0000","last_peered":"2023-11-08T08:18:44.272644+0000","last_clean":"2023-11-08T08:18:44.272644+0000","last_became_active":"2023-11-08T08:18:42.903182+0000","last_became_peered":"2023-11-08T08:18:42.903182+0000","last_unstale":"2023-11-08T08:18:44.272644+0000","last_undegraded":"2023-11-08T08:18:44.272644+0000","last_fullsized":"2023-11-08T08:18:44.272644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:23:54.970595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896906+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:18:42.896906+0000","last_peered":"2023-11-08T08:18:42.896906+0000","last_clean":"2023-11-08T08:18:42.896906+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:18:42.896906+0000","last_undegraded":"2023-11-08T08:18:42.896906+0000","last_fullsized":"2023-11-08T08:18:42.896906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272296+0000","last_change":"2023-11-08T08:18:42.506245+0000","last_active":"2023-11-08T08:18:44.272296+0000","last_peered":"2023-11-08T08:18:44.272296+0000","last_clean":"2023-11-08T08:18:44.272296+0000","last_became_active":"2023-11-08T08:18:42.505955+0000","last_became_peered":"2023-11-08T08:18:42.505955+0000","last_unstale":"2023-11-08T08:18:44.272296+0000","last_undegraded":"2023-11-08T08:18:44.272296+0000","last_fullsized":"2023-11-08T08:18:44.272296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272936+0000","last_change":"2023-11-08T08:18:42.905213+0000","last_active":"2023-11-08T08:18:44.272936+0000","last_peered":"2023-11-08T08:18:44.272936+0000","last_clean":"2023-11-08T08:18:44.272936+0000","last_became_active":"2023-11-08T08:18:42.904949+0000","last_became_peered":"2023-11-08T08:18:42.904949+0000","last_unstale":"2023-11-08T08:18:44.272936+0000","last_undegraded":"2023-11-08T08:18:44.272936+0000","last_fullsized":"2023-11-08T08:18:44.272936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:06.276090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.896829+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:18:42.896829+0000","last_peered":"2023-11-08T08:18:42.896829+0000","last_clean":"2023-11-08T08:18:42.896829+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:18:42.896829+0000","last_undegraded":"2023-11-08T08:18:42.896829+0000","last_fullsized":"2023-11-08T08:18:42.896829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272802+0000","last_change":"2023-11-08T08:18:42.502871+0000","last_active":"2023-11-08T08:18:44.272802+0000","last_peered":"2023-11-08T08:18:44.272802+0000","last_clean":"2023-11-08T08:18:44.272802+0000","last_became_active":"2023-11-08T08:18:42.502339+0000","last_became_peered":"2023-11-08T08:18:42.502339+0000","last_unstale":"2023-11-08T08:18:44.272802+0000","last_undegraded":"2023-11-08T08:18:44.272802+0000","last_fullsized":"2023-11-08T08:18:44.272802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:11:07.923861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272804+0000","last_change":"2023-11-08T08:18:42.910283+0000","last_active":"2023-11-08T08:18:44.272804+0000","last_peered":"2023-11-08T08:18:44.272804+0000","last_clean":"2023-11-08T08:18:44.272804+0000","last_became_active":"2023-11-08T08:18:42.909986+0000","last_became_peered":"2023-11-08T08:18:42.909986+0000","last_unstale":"2023-11-08T08:18:44.272804+0000","last_undegraded":"2023-11-08T08:18:44.272804+0000","last_fullsized":"2023-11-08T08:18:44.272804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-08T08:18:42.892038+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:18:42.892038+0000","last_peered":"2023-11-08T08:18:42.892038+0000","last_clean":"2023-11-08T08:18:42.892038+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:18:42.892038+0000","last_undegraded":"2023-11-08T08:18:42.892038+0000","last_fullsized":"2023-11-08T08:18:42.892038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893889+0000","last_change":"2023-11-08T08:18:42.503455+0000","last_active":"2023-11-08T08:18:43.893889+0000","last_peered":"2023-11-08T08:18:43.893889+0000","last_clean":"2023-11-08T08:18:43.893889+0000","last_became_active":"2023-11-08T08:18:42.503187+0000","last_became_peered":"2023-11-08T08:18:42.503187+0000","last_unstale":"2023-11-08T08:18:43.893889+0000","last_undegraded":"2023-11-08T08:18:43.893889+0000","last_fullsized":"2023-11-08T08:18:43.893889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:56:03.810805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272626+0000","last_change":"2023-11-08T08:18:42.911640+0000","last_active":"2023-11-08T08:18:44.272626+0000","last_peered":"2023-11-08T08:18:44.272626+0000","last_clean":"2023-11-08T08:18:44.272626+0000","last_became_active":"2023-11-08T08:18:42.911415+0000","last_became_peered":"2023-11-08T08:18:42.911415+0000","last_unstale":"2023-11-08T08:18:44.272626+0000","last_undegraded":"2023-11-08T08:18:44.272626+0000","last_fullsized":"2023-11-08T08:18:44.272626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:05:44.342444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272344+0000","last_change":"2023-11-08T08:18:42.501133+0000","last_active":"2023-11-08T08:18:44.272344+0000","last_peered":"2023-11-08T08:18:44.272344+0000","last_clean":"2023-11-08T08:18:44.272344+0000","last_became_active":"2023-11-08T08:18:42.500842+0000","last_became_peered":"2023-11-08T08:18:42.500842+0000","last_unstale":"2023-11-08T08:18:44.272344+0000","last_undegraded":"2023-11-08T08:18:44.272344+0000","last_fullsized":"2023-11-08T08:18:44.272344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:32:05.271102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272942+0000","last_change":"2023-11-08T08:18:42.905081+0000","last_active":"2023-11-08T08:18:44.272942+0000","last_peered":"2023-11-08T08:18:44.272942+0000","last_clean":"2023-11-08T08:18:44.272942+0000","last_became_active":"2023-11-08T08:18:42.904781+0000","last_became_peered":"2023-11-08T08:18:42.904781+0000","last_unstale":"2023-11-08T08:18:44.272942+0000","last_undegraded":"2023-11-08T08:18:44.272942+0000","last_fullsized":"2023-11-08T08:18:44.272942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:29:15.532692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898380+0000","last_change":"2023-11-08T08:18:42.505921+0000","last_active":"2023-11-08T08:18:43.898380+0000","last_peered":"2023-11-08T08:18:43.898380+0000","last_clean":"2023-11-08T08:18:43.898380+0000","last_became_active":"2023-11-08T08:18:42.505591+0000","last_became_peered":"2023-11-08T08:18:42.505591+0000","last_unstale":"2023-11-08T08:18:43.898380+0000","last_undegraded":"2023-11-08T08:18:43.898380+0000","last_fullsized":"2023-11-08T08:18:43.898380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:30:43.928582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893978+0000","last_change":"2023-11-08T08:18:42.907871+0000","last_active":"2023-11-08T08:18:43.893978+0000","last_peered":"2023-11-08T08:18:43.893978+0000","last_clean":"2023-11-08T08:18:43.893978+0000","last_became_active":"2023-11-08T08:18:42.907170+0000","last_became_peered":"2023-11-08T08:18:42.907170+0000","last_unstale":"2023-11-08T08:18:43.893978+0000","last_undegraded":"2023-11-08T08:18:43.893978+0000","last_fullsized":"2023-11-08T08:18:43.893978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:47:49.140236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272471+0000","last_change":"2023-11-08T08:18:42.504796+0000","last_active":"2023-11-08T08:18:44.272471+0000","last_peered":"2023-11-08T08:18:44.272471+0000","last_clean":"2023-11-08T08:18:44.272471+0000","last_became_active":"2023-11-08T08:18:42.504498+0000","last_became_peered":"2023-11-08T08:18:42.504498+0000","last_unstale":"2023-11-08T08:18:44.272471+0000","last_undegraded":"2023-11-08T08:18:44.272471+0000","last_fullsized":"2023-11-08T08:18:44.272471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:02:02.270931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893958+0000","last_change":"2023-11-08T08:18:42.906660+0000","last_active":"2023-11-08T08:18:43.893958+0000","last_peered":"2023-11-08T08:18:43.893958+0000","last_clean":"2023-11-08T08:18:43.893958+0000","last_became_active":"2023-11-08T08:18:42.906410+0000","last_became_peered":"2023-11-08T08:18:42.906410+0000","last_unstale":"2023-11-08T08:18:43.893958+0000","last_undegraded":"2023-11-08T08:18:43.893958+0000","last_fullsized":"2023-11-08T08:18:43.893958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:41:24.009486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899130+0000","last_change":"2023-11-08T08:18:42.497356+0000","last_active":"2023-11-08T08:18:43.899130+0000","last_peered":"2023-11-08T08:18:43.899130+0000","last_clean":"2023-11-08T08:18:43.899130+0000","last_became_active":"2023-11-08T08:18:42.497056+0000","last_became_peered":"2023-11-08T08:18:42.497056+0000","last_unstale":"2023-11-08T08:18:43.899130+0000","last_undegraded":"2023-11-08T08:18:43.899130+0000","last_fullsized":"2023-11-08T08:18:43.899130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:40:59.063729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272096+0000","last_change":"2023-11-08T08:18:42.905346+0000","last_active":"2023-11-08T08:18:44.272096+0000","last_peered":"2023-11-08T08:18:44.272096+0000","last_clean":"2023-11-08T08:18:44.272096+0000","last_became_active":"2023-11-08T08:18:42.905050+0000","last_became_peered":"2023-11-08T08:18:42.905050+0000","last_unstale":"2023-11-08T08:18:44.272096+0000","last_undegraded":"2023-11-08T08:18:44.272096+0000","last_fullsized":"2023-11-08T08:18:44.272096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899328+0000","last_change":"2023-11-08T08:18:42.499349+0000","last_active":"2023-11-08T08:18:43.899328+0000","last_peered":"2023-11-08T08:18:43.899328+0000","last_clean":"2023-11-08T08:18:43.899328+0000","last_became_active":"2023-11-08T08:18:42.499050+0000","last_became_peered":"2023-11-08T08:18:42.499050+0000","last_unstale":"2023-11-08T08:18:43.899328+0000","last_undegraded":"2023-11-08T08:18:43.899328+0000","last_fullsized":"2023-11-08T08:18:43.899328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:41.473595+0000","last_clean_scrub_stamp":"2023-11-08T08:18:41.473595+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:06:28.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.893668+0000","last_change":"2023-11-08T08:18:42.913391+0000","last_active":"2023-11-08T08:18:43.893668+0000","last_peered":"2023-11-08T08:18:43.893668+0000","last_clean":"2023-11-08T08:18:43.893668+0000","last_became_active":"2023-11-08T08:18:42.913124+0000","last_became_peered":"2023-11-08T08:18:42.913124+0000","last_unstale":"2023-11-08T08:18:43.893668+0000","last_undegraded":"2023-11-08T08:18:43.893668+0000","last_fullsized":"2023-11-08T08:18:43.893668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:11:39.516963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.898165+0000","last_change":"2023-11-08T08:18:42.912208+0000","last_active":"2023-11-08T08:18:43.898165+0000","last_peered":"2023-11-08T08:18:43.898165+0000","last_clean":"2023-11-08T08:18:43.898165+0000","last_became_active":"2023-11-08T08:18:42.911928+0000","last_became_peered":"2023-11-08T08:18:42.911928+0000","last_unstale":"2023-11-08T08:18:43.898165+0000","last_undegraded":"2023-11-08T08:18:43.898165+0000","last_fullsized":"2023-11-08T08:18:43.898165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:39:42.340496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.272255+0000","last_change":"2023-11-08T08:18:42.914191+0000","last_active":"2023-11-08T08:18:44.272255+0000","last_peered":"2023-11-08T08:18:44.272255+0000","last_clean":"2023-11-08T08:18:44.272255+0000","last_became_active":"2023-11-08T08:18:42.913919+0000","last_became_peered":"2023-11-08T08:18:42.913919+0000","last_unstale":"2023-11-08T08:18:44.272255+0000","last_undegraded":"2023-11-08T08:18:44.272255+0000","last_fullsized":"2023-11-08T08:18:44.272255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:13.095049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:44.271708+0000","last_change":"2023-11-08T08:18:42.900802+0000","last_active":"2023-11-08T08:18:44.271708+0000","last_peered":"2023-11-08T08:18:44.271708+0000","last_clean":"2023-11-08T08:18:44.271708+0000","last_became_active":"2023-11-08T08:18:42.900538+0000","last_became_peered":"2023-11-08T08:18:42.900538+0000","last_unstale":"2023-11-08T08:18:44.271708+0000","last_undegraded":"2023-11-08T08:18:44.271708+0000","last_fullsized":"2023-11-08T08:18:44.271708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:22:13.912166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899133+0000","last_change":"2023-11-08T08:18:42.906606+0000","last_active":"2023-11-08T08:18:43.899133+0000","last_peered":"2023-11-08T08:18:43.899133+0000","last_clean":"2023-11-08T08:18:43.899133+0000","last_became_active":"2023-11-08T08:18:42.906279+0000","last_became_peered":"2023-11-08T08:18:42.906279+0000","last_unstale":"2023-11-08T08:18:43.899133+0000","last_undegraded":"2023-11-08T08:18:43.899133+0000","last_fullsized":"2023-11-08T08:18:43.899133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:08:30.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-08T08:18:43.899035+0000","last_change":"2023-11-08T08:18:42.915689+0000","last_active":"2023-11-08T08:18:43.899035+0000","last_peered":"2023-11-08T08:18:43.899035+0000","last_clean":"2023-11-08T08:18:43.899035+0000","last_became_active":"2023-11-08T08:18:42.915121+0000","last_became_peered":"2023-11-08T08:18:42.915121+0000","last_unstale":"2023-11-08T08:18:43.899035+0000","last_undegraded":"2023-11-08T08:18:43.899035+0000","last_fullsized":"2023-11-08T08:18:43.899035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:42.478583+0000","last_clean_scrub_stamp":"2023-11-08T08:18:42.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:34:39.263663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7888,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363952,"statfs":{"total":96636764160,"available":96628686848,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":73265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":74582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7964,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363876,"statfs":{"total":96636764160,"available":96628609024,"internally_reserved":0,"allocated":217088,"data_stored":75984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9064,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362776,"statfs":{"total":96636764160,"available":96627482624,"internally_reserved":0,"allocated":819200,"data_stored":666456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7952,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363888,"statfs":{"total":96636764160,"available":96628621312,"internally_reserved":0,"allocated":204800,"data_stored":74436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9048,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362792,"statfs":{"total":96636764160,"available":96627499008,"internally_reserved":0,"allocated":802816,"data_stored":665370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:18:51.949 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-08T08:18:51.949 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-08T08:18:51.950 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-08T08:18:51.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-08T08:18:52.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.102+0000 7f707ce82700 1 Processor -- start 2023-11-08T08:18:52.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.102+0000 7f707ce82700 1 -- start start 2023-11-08T08:18:52.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707ce82700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f707815cc80 0x7f707806de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707ce82700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707806e3a0 0x7f7078070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707ce82700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70780b1810 con 0x7f707806a270 2023-11-08T08:18:52.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707ce82700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70780b2490 con 0x7f707815cc80 2023-11-08T08:18:52.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707ce82700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780b1af0 con 0x7f707806e3a0 2023-11-08T08:18:52.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f7075d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f707815cc80 0x7f707806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707659c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f707806a270 0x7f707806a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34388/0 (socket says 172.21.15.116:34388) 2023-11-08T08:18:52.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f707659c700 1 -- 172.21.15.116:0/1239297754 learned_addr learned my addr 172.21.15.116:0/1239297754 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:52.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f7076d9d700 1 --2- 172.21.15.116:0/1239297754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707806e3a0 0x7f7078070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f7075d9b700 1 -- 172.21.15.116:0/1239297754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707806e3a0 msgr2=0x7f7078070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f7075d9b700 1 --2- 172.21.15.116:0/1239297754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707806e3a0 0x7f7078070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f7075d9b700 1 -- 172.21.15.116:0/1239297754 >> v1:172.21.15.116:6789/0 conn(0x7f707806a270 legacy=0x7f707806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.103+0000 7f7075d9b700 1 -- 172.21.15.116:0/1239297754 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780b1dd0 con 0x7f707815cc80 2023-11-08T08:18:52.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f7075d9b700 1 --2- 172.21.15.116:0/1239297754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707815cc80 0x7f707806de60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f706c009840 tx=0x7f706c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ba2db6f73ae30 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707559a700 1 -- 172.21.15.116:0/1239297754 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c011830 con 0x7f707815cc80 2023-11-08T08:18:52.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707559a700 1 -- 172.21.15.116:0/1239297754 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c011990 con 0x7f707815cc80 2023-11-08T08:18:52.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707559a700 1 -- 172.21.15.116:0/1239297754 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c011d20 con 0x7f707815cc80 2023-11-08T08:18:52.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 -- 172.21.15.116:0/1239297754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707815cc80 msgr2=0x7f707806de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 --2- 172.21.15.116:0/1239297754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707815cc80 0x7f707806de60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f706c009840 tx=0x7f706c009b50 comp rx=0 tx=0).stop 2023-11-08T08:18:52.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 -- 172.21.15.116:0/1239297754 shutdown_connections 2023-11-08T08:18:52.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 --2- 172.21.15.116:0/1239297754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707806e3a0 0x7f7078070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 --2- 172.21.15.116:0/1239297754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707815cc80 0x7f707806de60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 -- 172.21.15.116:0/1239297754 >> 172.21.15.116:0/1239297754 conn(0x7f707806c010 msgr2=0x7f70780cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:52.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 -- 172.21.15.116:0/1239297754 shutdown_connections 2023-11-08T08:18:52.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.104+0000 7f707ce82700 1 -- 172.21.15.116:0/1239297754 wait complete. 2023-11-08T08:18:52.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 Processor -- start 2023-11-08T08:18:52.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 -- start start 2023-11-08T08:18:52.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 0x7f70780c3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f707806e3a0 0x7f7078152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707815cc80 0x7f707815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780c46b0 con 0x7f707806a270 2023-11-08T08:18:52.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780c3b90 con 0x7f707806e3a0 2023-11-08T08:18:52.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.106+0000 7f707ce82700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780c3e10 con 0x7f707815cc80 2023-11-08T08:18:52.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 0x7f70780c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 0x7f70780c3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39466/0 (socket says 172.21.15.116:39466) 2023-11-08T08:18:52.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 -- 172.21.15.116:0/419919778 learned_addr learned my addr 172.21.15.116:0/419919778 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:52.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f7076d9d700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707815cc80 0x7f707815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f7075d9b700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f707806e3a0 0x7f7078152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 -- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707815cc80 msgr2=0x7f707815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707815cc80 0x7f707815edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 -- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f707806e3a0 msgr2=0x7f7078152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f707806e3a0 0x7f7078152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 -- 172.21.15.116:0/419919778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0094a0 con 0x7f707806a270 2023-11-08T08:18:52.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707659c700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 0x7f70780c3650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f706800c430 tx=0x7f706800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068005b70 con 0x7f707806a270 2023-11-08T08:18:52.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7068005cd0 con 0x7f707806a270 2023-11-08T08:18:52.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707806e800 con 0x7f707806a270 2023-11-08T08:18:52.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680072f0 con 0x7f707806a270 2023-11-08T08:18:52.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.107+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70780b20b0 con 0x7f707806a270 2023-11-08T08:18:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.108+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707815daa0 con 0x7f707806a270 2023-11-08T08:18:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.109+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7068019030 con 0x7f707806a270 2023-11-08T08:18:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.109+0000 7f7066ffd700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f70580682f0 0x7f705806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.109+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7068084e30 con 0x7f707806a270 2023-11-08T08:18:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.109+0000 7f7075d9b700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f70580682f0 0x7f705806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.110+0000 7f7075d9b700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f70580682f0 0x7f705806a7a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f706c0090b0 tx=0x7f706c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.113+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7068051690 con 0x7f707806a270 2023-11-08T08:18:52.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.338+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70780743f0 con 0x7f707806a270 2023-11-08T08:18:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.338+0000 7f7066ffd700 1 -- 172.21.15.116:0/419919778 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7068017080 con 0x7f707806a270 2023-11-08T08:18:52.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:52.340 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-08T08:18:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f70580682f0 msgr2=0x7f705806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f70580682f0 0x7f705806a7a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f706c0090b0 tx=0x7f706c008030 comp rx=0 tx=0).stop 2023-11-08T08:18:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 msgr2=0x7f70780c3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 0x7f70780c3650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f706800c430 tx=0x7f706800c740 comp rx=0 tx=0).stop 2023-11-08T08:18:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 shutdown_connections 2023-11-08T08:18:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f707815cc80 0x7f707815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707806a270 0x7f70780c3650 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f70580682f0 0x7f705806a7a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 --2- 172.21.15.116:0/419919778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f707806e3a0 0x7f7078152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 >> 172.21.15.116:0/419919778 conn(0x7f707806c010 msgr2=0x7f70780cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 shutdown_connections 2023-11-08T08:18:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.341+0000 7f707ce82700 1 -- 172.21.15.116:0/419919778 wait complete. 2023-11-08T08:18:52.354 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-08T08:18:52.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:18:52.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.473+0000 7f35bd51a700 1 Processor -- start 2023-11-08T08:18:52.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.473+0000 7f35bd51a700 1 -- start start 2023-11-08T08:18:52.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35bd51a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b806a980 0x7f35b806dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35bd51a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35b806e110 0x7f35b80ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35bd51a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b8070bf0 con 0x7f35b815c640 2023-11-08T08:18:52.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35bd51a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b806c880 con 0x7f35b806e110 2023-11-08T08:18:52.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35bd51a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b806cad0 con 0x7f35b806a980 2023-11-08T08:18:52.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35b77fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35b806e110 0x7f35b80ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35b77fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35b806e110 0x7f35b80ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58672/0 (socket says 172.21.15.116:58672) 2023-11-08T08:18:52.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.474+0000 7f35b77fe700 1 -- 172.21.15.116:0/1123219283 learned_addr learned my addr 172.21.15.116:0/1123219283 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:52.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b67fc700 1 --2- 172.21.15.116:0/1123219283 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b806a980 0x7f35b806dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b77fe700 1 -- 172.21.15.116:0/1123219283 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b806a980 msgr2=0x7f35b806dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b77fe700 1 --2- 172.21.15.116:0/1123219283 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b806a980 0x7f35b806dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b77fe700 1 -- 172.21.15.116:0/1123219283 >> v1:172.21.15.129:6789/0 conn(0x7f35b815c640 legacy=0x7f35b806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b77fe700 1 -- 172.21.15.116:0/1123219283 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b806cdb0 con 0x7f35b806e110 2023-11-08T08:18:52.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b77fe700 1 --2- 172.21.15.116:0/1123219283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b80ce8b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009e50 tx=0x7f35b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e443e7aebb4ca51 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b5ffb700 1 -- 172.21.15.116:0/1123219283 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000e070 con 0x7f35b806e110 2023-11-08T08:18:52.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b5ffb700 1 -- 172.21.15.116:0/1123219283 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b0003a20 con 0x7f35b806e110 2023-11-08T08:18:52.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35b5ffb700 1 -- 172.21.15.116:0/1123219283 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0003dd0 con 0x7f35b806e110 2023-11-08T08:18:52.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 -- 172.21.15.116:0/1123219283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 msgr2=0x7f35b80ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1123219283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b80ce8b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009e50 tx=0x7f35b000b330 comp rx=0 tx=0).stop 2023-11-08T08:18:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 -- 172.21.15.116:0/1123219283 shutdown_connections 2023-11-08T08:18:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1123219283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b80ce8b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1123219283 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b806a980 0x7f35b806dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 -- 172.21.15.116:0/1123219283 >> 172.21.15.116:0/1123219283 conn(0x7f35b806bfd0 msgr2=0x7f35b80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:52.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 -- 172.21.15.116:0/1123219283 shutdown_connections 2023-11-08T08:18:52.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.475+0000 7f35bd51a700 1 -- 172.21.15.116:0/1123219283 wait complete. 2023-11-08T08:18:52.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 Processor -- start 2023-11-08T08:18:52.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 -- start start 2023-11-08T08:18:52.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35b806a980 0x7f35b814e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b81f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b815c640 0x7f35b81549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8155210 con 0x7f35b806e110 2023-11-08T08:18:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81554f0 con 0x7f35b806a980 2023-11-08T08:18:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.477+0000 7f35bd51a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81557d0 con 0x7f35b815c640 2023-11-08T08:18:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b81f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b81f6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58682/0 (socket says 172.21.15.116:58682) 2023-11-08T08:18:52.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 -- 172.21.15.116:0/1392459968 learned_addr learned my addr 172.21.15.116:0/1392459968 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:52.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b6ffd700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35b806a980 0x7f35b814e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b77fe700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b815c640 0x7f35b81549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 -- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b815c640 msgr2=0x7f35b81549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b815c640 0x7f35b81549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 -- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35b806a980 msgr2=0x7f35b814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35b806a980 0x7f35b814e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 -- 172.21.15.116:0/1392459968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0009b30 con 0x7f35b806e110 2023-11-08T08:18:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b6ffd700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35b806a980 0x7f35b814e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:52.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b77fe700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b815c640 0x7f35b81549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:52.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35b67fc700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b81f6620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f35a800c7f0 tx=0x7f35a800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8011800 con 0x7f35b806e110 2023-11-08T08:18:52.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8011960 con 0x7f35b806e110 2023-11-08T08:18:52.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b806d370 con 0x7f35b806e110 2023-11-08T08:18:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8011d10 con 0x7f35b806e110 2023-11-08T08:18:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.478+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b806d090 con 0x7f35b806e110 2023-11-08T08:18:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.479+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b806d650 con 0x7f35b806e110 2023-11-08T08:18:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.479+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f35a8021070 con 0x7f35b806e110 2023-11-08T08:18:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.480+0000 7f359f7fe700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a40682f0 0x7f35a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.480+0000 7f35b6ffd700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a40682f0 0x7f35a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.480+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f35a8085a10 con 0x7f35b806e110 2023-11-08T08:18:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.480+0000 7f35b6ffd700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a40682f0 0x7f35a406a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35ac001750 tx=0x7f35ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.484+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35a8052270 con 0x7f35b806e110 2023-11-08T08:18:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.664+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35b80c3700 con 0x7f35b806e110 2023-11-08T08:18:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.666+0000 7f359f7fe700 1 -- 172.21.15.116:0/1392459968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f35a8055890 con 0x7f35b806e110 2023-11-08T08:18:52.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:18:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a40682f0 msgr2=0x7f35a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a40682f0 0x7f35a406a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35ac001750 tx=0x7f35ac009040 comp rx=0 tx=0).stop 2023-11-08T08:18:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 msgr2=0x7f35b81f6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b81f6620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f35a800c7f0 tx=0x7f35a800cbb0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 shutdown_connections 2023-11-08T08:18:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35b815c640 0x7f35b81549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a40682f0 0x7f35a406a7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35b806a980 0x7f35b814e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 --2- 172.21.15.116:0/1392459968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35b806e110 0x7f35b81f6620 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 >> 172.21.15.116:0/1392459968 conn(0x7f35b806bfd0 msgr2=0x7f35b80adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 shutdown_connections 2023-11-08T08:18:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.668+0000 7f35bd51a700 1 -- 172.21.15.116:0/1392459968 wait complete. 2023-11-08T08:18:52.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:18:52.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:18:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.878+0000 7fe157e92700 1 Processor -- start 2023-11-08T08:18:52.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.878+0000 7fe157e92700 1 -- start start 2023-11-08T08:18:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.879+0000 7fe157e92700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe1500b1650 0x7fe1500b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.879+0000 7fe157e92700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe150159ec0 con 0x7fe1501599b0 2023-11-08T08:18:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.879+0000 7fe157e92700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15015aa60 con 0x7fe1500af0a0 2023-11-08T08:18:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.879+0000 7fe157e92700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1500b1f60 con 0x7fe1500b1650 2023-11-08T08:18:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.879+0000 7fe15542d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe1501599b0 0x7fe15015d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51654/0 (socket says 172.21.15.116:51654) 2023-11-08T08:18:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.879+0000 7fe15542d700 1 -- 172.21.15.116:0/2075576515 learned_addr learned my addr 172.21.15.116:0/2075576515 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256292310 0 0) 0x7fe150159ec0 con 0x7fe1501599b0 2023-11-08T08:18:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13c0035b0 con 0x7fe1501599b0 2023-11-08T08:18:52.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe155c2e700 1 --2- 172.21.15.116:0/2075576515 >> v2:172.21.15.129:3300/0 conn(0x7fe1500b1650 0x7fe1500b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1639256371 0 0) 0x7fe13c0035b0 con 0x7fe1501599b0 2023-11-08T08:18:52.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 >> v2:172.21.15.129:3300/0 conn(0x7fe1500b1650 msgr2=0x7fe1500b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 --2- 172.21.15.116:0/2075576515 >> v2:172.21.15.129:3300/0 conn(0x7fe1500b1650 0x7fe1500b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 >> v1:172.21.15.129:6789/0 conn(0x7fe1500af0a0 legacy=0x7fe15015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15015d8a0 con 0x7fe1501599b0 2023-11-08T08:18:52.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fe14c001670 con 0x7fe1501599b0 2023-11-08T08:18:52.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 >> v1:172.21.15.116:6789/0 conn(0x7fe1501599b0 legacy=0x7fe15015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c003a00 0x7fe13c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c004850 0x7fe13c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13c005200 0x7fe13c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500b1f60 con 0x7fe13c005200 2023-11-08T08:18:52.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c00c110 con 0x7fe13c003a00 2023-11-08T08:18:52.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c00c2d0 con 0x7fe13c004850 2023-11-08T08:18:52.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13c005200 0x7fe13c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe155c2e700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c003a00 0x7fe13c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15542d700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c004850 0x7fe13c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 -- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c004850 msgr2=0x7fe13c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c004850 0x7fe13c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 -- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c003a00 msgr2=0x7fe13c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c003a00 0x7fe13c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 -- 172.21.15.116:0/2075576515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0035b0 con 0x7fe13c005200 2023-11-08T08:18:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 -- 172.21.15.116:0/2075576515 reap_dead start 2023-11-08T08:18:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.880+0000 7fe15642f700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13c005200 0x7fe13c00ba10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe1480049e0 tx=0x7fe13c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450ce6c68fd3964e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148004450 con 0x7fe13c005200 2023-11-08T08:18:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14800cc50 con 0x7fe13c005200 2023-11-08T08:18:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe154c2c700 1 -- 172.21.15.116:0/2075576515 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148004a10 con 0x7fe13c005200 2023-11-08T08:18:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 -- 172.21.15.116:0/2075576515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13c005200 msgr2=0x7fe13c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13c005200 0x7fe13c00ba10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe1480049e0 tx=0x7fe13c005880 comp rx=0 tx=0).stop 2023-11-08T08:18:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 -- 172.21.15.116:0/2075576515 shutdown_connections 2023-11-08T08:18:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 --2- 172.21.15.116:0/2075576515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13c005200 0x7fe13c00ba10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 -- 172.21.15.116:0/2075576515 >> 172.21.15.116:0/2075576515 conn(0x7fe15006c010 msgr2=0x7fe1500cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 -- 172.21.15.116:0/2075576515 shutdown_connections 2023-11-08T08:18:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.881+0000 7fe157e92700 1 -- 172.21.15.116:0/2075576515 wait complete. 2023-11-08T08:18:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.883+0000 7fe157e92700 1 Processor -- start 2023-11-08T08:18:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.883+0000 7fe157e92700 1 -- start start 2023-11-08T08:18:52.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c005200 0x7fe1500af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 0x7fe150147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1501e4eb0 0x7fe1501e5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501e6710 con 0x7fe150146bd0 2023-11-08T08:18:52.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe150147580 con 0x7fe1501e4eb0 2023-11-08T08:18:52.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501e5840 con 0x7fe13c005200 2023-11-08T08:18:52.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 0x7fe150147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 0x7fe150147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58702/0 (socket says 172.21.15.116:58702) 2023-11-08T08:18:52.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 -- 172.21.15.116:0/1720061384 learned_addr learned my addr 172.21.15.116:0/1720061384 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:52.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe155c2e700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c005200 0x7fe1500af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15642f700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1501e4eb0 0x7fe1501e5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 -- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c005200 msgr2=0x7fe1500af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c005200 0x7fe1500af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 -- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1501e4eb0 msgr2=0x7fe1501e5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:52.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1501e4eb0 0x7fe1501e5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:52.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 -- 172.21.15.116:0/1720061384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15015de60 con 0x7fe150146bd0 2023-11-08T08:18:52.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe15542d700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 0x7fe150147040 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00eb20 tx=0x7fe14c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c00c5d0 con 0x7fe150146bd0 2023-11-08T08:18:52.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c00c730 con 0x7fe150146bd0 2023-11-08T08:18:52.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15015db80 con 0x7fe150146bd0 2023-11-08T08:18:52.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c00cae0 con 0x7fe150146bd0 2023-11-08T08:18:52.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.884+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1500b1a20 con 0x7fe150146bd0 2023-11-08T08:18:52.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.885+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe150154d20 con 0x7fe150146bd0 2023-11-08T08:18:52.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.886+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe14c00cc40 con 0x7fe150146bd0 2023-11-08T08:18:52.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.886+0000 7fe1467fc700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe1340682f0 0x7fe13406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:52.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.886+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe14c085b10 con 0x7fe150146bd0 2023-11-08T08:18:52.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.886+0000 7fe155c2e700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe1340682f0 0x7fe13406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:52.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.886+0000 7fe155c2e700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe1340682f0 0x7fe13406a7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe140009980 tx=0x7fe140008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:52.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:52.890+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe150154d20 con 0x7fe150146bd0 2023-11-08T08:18:53.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.071+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe15006e620 con 0x7fe150146bd0 2023-11-08T08:18:53.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.071+0000 7fe1467fc700 1 -- 172.21.15.116:0/1720061384 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe14c0522e0 con 0x7fe150146bd0 2023-11-08T08:18:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:18:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe1340682f0 msgr2=0x7fe13406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe1340682f0 0x7fe13406a7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe140009980 tx=0x7fe140008040 comp rx=0 tx=0).stop 2023-11-08T08:18:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 msgr2=0x7fe150147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 0x7fe150147040 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00eb20 tx=0x7fe14c00ee30 comp rx=0 tx=0).stop 2023-11-08T08:18:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 shutdown_connections 2023-11-08T08:18:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe1340682f0 0x7fe13406a7a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1501e4eb0 0x7fe1501e5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe13c005200 0x7fe1500af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 --2- 172.21.15.116:0/1720061384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe150146bd0 0x7fe150147040 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 >> 172.21.15.116:0/1720061384 conn(0x7fe15006c010 msgr2=0x7fe1500cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:53.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 shutdown_connections 2023-11-08T08:18:53.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.073+0000 7fe157e92700 1 -- 172.21.15.116:0/1720061384 wait complete. 2023-11-08T08:18:53.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:18:53.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.208+0000 7f0393172700 1 Processor -- start 2023-11-08T08:18:53.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.208+0000 7f0393172700 1 -- start start 2023-11-08T08:18:53.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f0393172700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f038c0cf710 0x7f038c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f0393172700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c06a980 0x7f038c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f0393172700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038c15cc10 con 0x7f038c071890 2023-11-08T08:18:53.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f0393172700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038c06c9b0 con 0x7f038c0cf710 2023-11-08T08:18:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f0393172700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c06cc90 con 0x7f038c06a980 2023-11-08T08:18:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f039170f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c06a980 0x7f038c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f0390f0e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f038c0cf710 0x7f038c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f039170f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c06a980 0x7f038c071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:41790/0 (socket says 172.21.15.116:41790) 2023-11-08T08:18:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f039170f700 1 -- 172.21.15.116:0/3577483392 learned_addr learned my addr 172.21.15.116:0/3577483392 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.209+0000 7f038bfff700 1 --1- 172.21.15.116:0/3577483392 >> v1:172.21.15.129:6789/0 conn(0x7f038c071890 0x7f038c06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:53860/0 (socket says 172.21.15.116:53860) 2023-11-08T08:18:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0390f0e700 1 -- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c06a980 msgr2=0x7f038c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0390f0e700 1 --2- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c06a980 0x7f038c071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0390f0e700 1 -- 172.21.15.116:0/3577483392 >> v1:172.21.15.129:6789/0 conn(0x7f038c071890 legacy=0x7f038c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0390f0e700 1 -- 172.21.15.116:0/3577483392 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c06cf70 con 0x7f038c0cf710 2023-11-08T08:18:53.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f038b7fe700 1 -- 172.21.15.116:0/3577483392 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039132670 0 0) 0x7f038c15cc10 con 0x7f038c071890 2023-11-08T08:18:53.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0390f0e700 1 --2- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0cf710 0x7f038c0ce740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0380009870 tx=0x7f0380009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74b763e615235273 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f038b7fe700 1 -- 172.21.15.116:0/3577483392 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380011830 con 0x7f038c0cf710 2023-11-08T08:18:53.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f038b7fe700 1 -- 172.21.15.116:0/3577483392 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0380011990 con 0x7f038c0cf710 2023-11-08T08:18:53.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f038b7fe700 1 -- 172.21.15.116:0/3577483392 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380011d20 con 0x7f038c0cf710 2023-11-08T08:18:53.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0393172700 1 -- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0cf710 msgr2=0x7f038c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.210+0000 7f0393172700 1 --2- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0cf710 0x7f038c0ce740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0380009870 tx=0x7f0380009b80 comp rx=0 tx=0).stop 2023-11-08T08:18:53.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.211+0000 7f0393172700 1 -- 172.21.15.116:0/3577483392 shutdown_connections 2023-11-08T08:18:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.211+0000 7f0393172700 1 --2- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0cf710 0x7f038c0ce740 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.211+0000 7f0393172700 1 --2- 172.21.15.116:0/3577483392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c06a980 0x7f038c071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.211+0000 7f0393172700 1 -- 172.21.15.116:0/3577483392 >> 172.21.15.116:0/3577483392 conn(0x7f038c06c010 msgr2=0x7f038c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.211+0000 7f0393172700 1 -- 172.21.15.116:0/3577483392 shutdown_connections 2023-11-08T08:18:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.211+0000 7f0393172700 1 -- 172.21.15.116:0/3577483392 wait complete. 2023-11-08T08:18:53.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.212+0000 7f0393172700 1 Processor -- start 2023-11-08T08:18:53.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.212+0000 7f0393172700 1 -- start start 2023-11-08T08:18:53.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 0x7f038c0afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c071890 0x7f038c1e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c0cf710 0x7f038c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c1e70e0 con 0x7f038c06a980 2023-11-08T08:18:53.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0b0240 con 0x7f038c071890 2023-11-08T08:18:53.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0b0520 con 0x7f038c0cf710 2023-11-08T08:18:53.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 0x7f038c0afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 0x7f038c0afbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58712/0 (socket says 172.21.15.116:58712) 2023-11-08T08:18:53.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 -- 172.21.15.116:0/2179046887 learned_addr learned my addr 172.21.15.116:0/2179046887 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:53.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f038bfff700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c071890 0x7f038c1e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f039170f700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c0cf710 0x7f038c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 -- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c0cf710 msgr2=0x7f038c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c0cf710 0x7f038c0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 -- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c071890 msgr2=0x7f038c1e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c071890 0x7f038c1e6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 -- 172.21.15.116:0/2179046887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03800094a0 con 0x7f038c06a980 2023-11-08T08:18:53.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0390f0e700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 0x7f038c0afbb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0380014040 tx=0x7f0380004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038001a8b0 con 0x7f038c06a980 2023-11-08T08:18:53.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038001aa10 con 0x7f038c06a980 2023-11-08T08:18:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038c06d530 con 0x7f038c06a980 2023-11-08T08:18:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038001ada0 con 0x7f038c06a980 2023-11-08T08:18:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.213+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038c06d250 con 0x7f038c06a980 2023-11-08T08:18:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.214+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038c160480 con 0x7f038c06a980 2023-11-08T08:18:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.215+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f038000f050 con 0x7f038c06a980 2023-11-08T08:18:53.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.215+0000 7f03897fa700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f03700682f0 0x7f037006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.215+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0380086ec0 con 0x7f038c06a980 2023-11-08T08:18:53.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.215+0000 7f038bfff700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f03700682f0 0x7f037006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.216+0000 7f038bfff700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f03700682f0 0x7f037006a7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0378007b10 tx=0x7f0378007a60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.220+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0380053720 con 0x7f038c06a980 2023-11-08T08:18:53.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.399+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f038c0743f0 con 0x7f038c06a980 2023-11-08T08:18:53.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.400+0000 7f03897fa700 1 -- 172.21.15.116:0/2179046887 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0380015070 con 0x7f038c06a980 2023-11-08T08:18:53.401 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:18:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.401+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f03700682f0 msgr2=0x7f037006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.401+0000 7f0393172700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f03700682f0 0x7f037006a7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0378007b10 tx=0x7f0378007a60 comp rx=0 tx=0).stop 2023-11-08T08:18:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.401+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 msgr2=0x7f038c0afbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.401+0000 7f0393172700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 0x7f038c0afbb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0380014040 tx=0x7f0380004750 comp rx=0 tx=0).stop 2023-11-08T08:18:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 shutdown_connections 2023-11-08T08:18:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f038c0cf710 0x7f038c0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038c06a980 0x7f038c0afbb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f03700682f0 0x7f037006a7a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 --2- 172.21.15.116:0/2179046887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c071890 0x7f038c1e6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 >> 172.21.15.116:0/2179046887 conn(0x7f038c06c010 msgr2=0x7f038c15c150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 shutdown_connections 2023-11-08T08:18:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.402+0000 7f0393172700 1 -- 172.21.15.116:0/2179046887 wait complete. 2023-11-08T08:18:53.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:18:53.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.536+0000 7fa7b77a3700 1 Processor -- start 2023-11-08T08:18:53.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.537+0000 7fa7b77a3700 1 -- start start 2023-11-08T08:18:53.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b77a3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7b00b1650 0x7fa7b00b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b77a3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00af0a0 0x7fa7b015a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b77a3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b0159ec0 con 0x7fa7b01599b0 2023-11-08T08:18:53.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b77a3700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b00b1f60 con 0x7fa7b00b1650 2023-11-08T08:18:53.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b77a3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b015d8a0 con 0x7fa7b00af0a0 2023-11-08T08:18:53.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7b00b1650 0x7fa7b00b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7b00b1650 0x7fa7b00b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58718/0 (socket says 172.21.15.116:58718) 2023-11-08T08:18:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 -- 172.21.15.116:0/2229565285 learned_addr learned my addr 172.21.15.116:0/2229565285 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b5d40700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00af0a0 0x7fa7b015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 -- 172.21.15.116:0/2229565285 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00af0a0 msgr2=0x7fa7b015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00af0a0 0x7fa7b015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 -- 172.21.15.116:0/2229565285 >> v1:172.21.15.129:6789/0 conn(0x7fa7b01599b0 legacy=0x7fa7b015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 -- 172.21.15.116:0/2229565285 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b015db80 con 0x7fa7b00b1650 2023-11-08T08:18:53.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b5d40700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00af0a0 0x7fa7b015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:53.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.538+0000 7fa7b553f700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00b1650 0x7fa7b00b1a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0098f0 tx=0x7fa7ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43e66442861b571 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7a7fff700 1 -- 172.21.15.116:0/2229565285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac004230 con 0x7fa7b00b1650 2023-11-08T08:18:53.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7a7fff700 1 -- 172.21.15.116:0/2229565285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac004390 con 0x7fa7b00b1650 2023-11-08T08:18:53.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7a7fff700 1 -- 172.21.15.116:0/2229565285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0036a0 con 0x7fa7b00b1650 2023-11-08T08:18:53.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2229565285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00b1650 msgr2=0x7fa7b00b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00b1650 0x7fa7b00b1a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0098f0 tx=0x7fa7ac009c00 comp rx=0 tx=0).stop 2023-11-08T08:18:53.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2229565285 shutdown_connections 2023-11-08T08:18:53.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00b1650 0x7fa7b00b1a20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2229565285 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00af0a0 0x7fa7b015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2229565285 >> 172.21.15.116:0/2229565285 conn(0x7fa7b006c010 msgr2=0x7fa7b00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:53.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2229565285 shutdown_connections 2023-11-08T08:18:53.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.539+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2229565285 wait complete. 2023-11-08T08:18:53.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.540+0000 7fa7b77a3700 1 Processor -- start 2023-11-08T08:18:53.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.540+0000 7fa7b77a3700 1 -- start start 2023-11-08T08:18:53.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b77a3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 0x7fa7b006a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b77a3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00b1650 0x7fa7b00ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b77a3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b01f6080 0x7fa7b00adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b77a3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00b3400 con 0x7fa7b00af0a0 2023-11-08T08:18:53.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b77a3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00ae1e0 con 0x7fa7b01f6080 2023-11-08T08:18:53.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b77a3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00c39c0 con 0x7fa7b00b1650 2023-11-08T08:18:53.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 0x7fa7b006a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 0x7fa7b006a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58720/0 (socket says 172.21.15.116:58720) 2023-11-08T08:18:53.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 -- 172.21.15.116:0/2658689618 learned_addr learned my addr 172.21.15.116:0/2658689618 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:53.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b4d3e700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00b1650 0x7fa7b00ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b5d40700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b01f6080 0x7fa7b00adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 -- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00b1650 msgr2=0x7fa7b00ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00b1650 0x7fa7b00ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 -- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b01f6080 msgr2=0x7fa7b00adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b01f6080 0x7fa7b00adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b553f700 1 -- 172.21.15.116:0/2658689618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac009550 con 0x7fa7b00af0a0 2023-11-08T08:18:53.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b4d3e700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00b1650 0x7fa7b00ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:53.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.541+0000 7fa7b5d40700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b01f6080 0x7fa7b00adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:53.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.542+0000 7fa7b553f700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 0x7fa7b006a4e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009130 tx=0x7fa7ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.542+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00b860 con 0x7fa7b00af0a0 2023-11-08T08:18:53.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.542+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac00b9c0 con 0x7fa7b00af0a0 2023-11-08T08:18:53.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.542+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b015a890 con 0x7fa7b00af0a0 2023-11-08T08:18:53.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.542+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00bd70 con 0x7fa7b00af0a0 2023-11-08T08:18:53.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.542+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b015de60 con 0x7fa7b00af0a0 2023-11-08T08:18:53.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.543+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b015aa20 con 0x7fa7b00af0a0 2023-11-08T08:18:53.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.543+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7ac017020 con 0x7fa7b00af0a0 2023-11-08T08:18:53.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.544+0000 7fa7a5ffb700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7940682f0 0x7fa79406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.544+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa7ac087640 con 0x7fa7b00af0a0 2023-11-08T08:18:53.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.544+0000 7fa7b4d3e700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7940682f0 0x7fa79406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.544+0000 7fa7b4d3e700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7940682f0 0x7fa79406a7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa7b01f5fe0 tx=0x7fa7a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.548+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7ac053db0 con 0x7fa7b00af0a0 2023-11-08T08:18:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.729+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa7b00b3540 con 0x7fa7b00af0a0 2023-11-08T08:18:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.729+0000 7fa7a5ffb700 1 -- 172.21.15.116:0/2658689618 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7ac0573d0 con 0x7fa7b00af0a0 2023-11-08T08:18:53.731 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:18:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7940682f0 msgr2=0x7fa79406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7940682f0 0x7fa79406a7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa7b01f5fe0 tx=0x7fa7a0008040 comp rx=0 tx=0).stop 2023-11-08T08:18:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 msgr2=0x7fa7b006a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 0x7fa7b006a4e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009130 tx=0x7fa7ac004750 comp rx=0 tx=0).stop 2023-11-08T08:18:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 shutdown_connections 2023-11-08T08:18:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa7940682f0 0x7fa79406a7a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b01f6080 0x7fa7b00adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7b00af0a0 0x7fa7b006a4e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 --2- 172.21.15.116:0/2658689618 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7b00b1650 0x7fa7b00ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 >> 172.21.15.116:0/2658689618 conn(0x7fa7b006c010 msgr2=0x7fa7b015b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:53.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 shutdown_connections 2023-11-08T08:18:53.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.731+0000 7fa7b77a3700 1 -- 172.21.15.116:0/2658689618 wait complete. 2023-11-08T08:18:53.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:18:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.866+0000 7f718bfd4700 1 Processor -- start 2023-11-08T08:18:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.866+0000 7f718bfd4700 1 -- start start 2023-11-08T08:18:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718bfd4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f718406d170 0x7f71840b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718bfd4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 0x7f71840b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718bfd4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7184161370 con 0x7f718415fe50 2023-11-08T08:18:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718bfd4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718406d5c0 con 0x7f718406d170 2023-11-08T08:18:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718bfd4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840b3190 con 0x7f71840b27d0 2023-11-08T08:18:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f7189d70700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f718415fe50 0x7f7184160ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51666/0 (socket says 172.21.15.116:51666) 2023-11-08T08:18:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f7189d70700 1 -- 172.21.15.116:0/1353601235 learned_addr learned my addr 172.21.15.116:0/1353601235 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718a571700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 0x7f71840b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137440346 0 0) 0x7f7184161370 con 0x7f718415fe50 2023-11-08T08:18:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f718956f700 1 --2- 172.21.15.116:0/1353601235 >> v2:172.21.15.129:3300/0 conn(0x7f718406d170 0x7f71840b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.867+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71700035b0 con 0x7f718415fe50 2023-11-08T08:18:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1493730018 0 0) 0x7f71700035b0 con 0x7f718415fe50 2023-11-08T08:18:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 msgr2=0x7f71840b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 0x7f71840b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 >> v2:172.21.15.129:3300/0 conn(0x7f718406d170 msgr2=0x7f71840b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 --2- 172.21.15.116:0/1353601235 >> v2:172.21.15.129:3300/0 conn(0x7f718406d170 0x7f71840b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71840cf700 con 0x7f718415fe50 2023-11-08T08:18:53.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f7174001670 con 0x7f718415fe50 2023-11-08T08:18:53.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 >> v1:172.21.15.116:6789/0 conn(0x7f718415fe50 legacy=0x7f7184160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 0x7f7170003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f7170004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7170004d90 0x7f7170005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840b3190 con 0x7f7170003f80 2023-11-08T08:18:53.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717000ba10 con 0x7f7170004d90 2023-11-08T08:18:53.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717000bb50 con 0x7f71840b27d0 2023-11-08T08:18:53.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f7170004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7189d70700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 0x7f7170003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718a571700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7170004d90 0x7f7170005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 -- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 msgr2=0x7f7170003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71840b27d0 0x7f7170003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 -- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7170004d90 msgr2=0x7f7170005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7170004d90 0x7f7170005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 -- 172.21.15.116:0/1353601235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700035b0 con 0x7f7170003f80 2023-11-08T08:18:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f718956f700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f7170004850 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7180009b60 tx=0x7f7170005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d88c90d4873f70 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718000e6c0 con 0x7f7170003f80 2023-11-08T08:18:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718000e820 con 0x7f7170003f80 2023-11-08T08:18:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.868+0000 7f7188d6e700 1 -- 172.21.15.116:0/1353601235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718000eb90 con 0x7f7170003f80 2023-11-08T08:18:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718bfd4700 1 -- 172.21.15.116:0/1353601235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 msgr2=0x7f7170004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718bfd4700 1 --2- 172.21.15.116:0/1353601235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f7170004850 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7180009b60 tx=0x7f7170005880 comp rx=0 tx=0).stop 2023-11-08T08:18:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718a571700 1 -- 172.21.15.116:0/1353601235 reap_dead start 2023-11-08T08:18:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718bfd4700 1 -- 172.21.15.116:0/1353601235 shutdown_connections 2023-11-08T08:18:53.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718bfd4700 1 -- 172.21.15.116:0/1353601235 >> 172.21.15.116:0/1353601235 conn(0x7f718406ea80 msgr2=0x7f718406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:53.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718bfd4700 1 -- 172.21.15.116:0/1353601235 shutdown_connections 2023-11-08T08:18:53.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.869+0000 7f718bfd4700 1 -- 172.21.15.116:0/1353601235 wait complete. 2023-11-08T08:18:53.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 Processor -- start 2023-11-08T08:18:53.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 -- start start 2023-11-08T08:18:53.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f71840c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71840cbea0 0x7f71840c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71841f1ff0 0x7f71841f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840c7610 con 0x7f7170003f80 2023-11-08T08:18:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840cd470 con 0x7f71840cbea0 2023-11-08T08:18:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.871+0000 7f718bfd4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840cc3d0 con 0x7f71841f1ff0 2023-11-08T08:18:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f71840c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f71840c6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58732/0 (socket says 172.21.15.116:58732) 2023-11-08T08:18:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 -- 172.21.15.116:0/62178084 learned_addr learned my addr 172.21.15.116:0/62178084 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f718956f700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71840cbea0 0x7f71840c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f718a571700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71841f1ff0 0x7f71841f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 -- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71841f1ff0 msgr2=0x7f71841f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71841f1ff0 0x7f71841f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 -- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71840cbea0 msgr2=0x7f71840c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71840cbea0 0x7f71840c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 -- 172.21.15.116:0/62178084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71840cf0b0 con 0x7f7170003f80 2023-11-08T08:18:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f7189d70700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f71840c6b90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7174007310 tx=0x7f7174007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717400f6c0 con 0x7f7170003f80 2023-11-08T08:18:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717400f820 con 0x7f7170003f80 2023-11-08T08:18:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71840cedd0 con 0x7f7170003f80 2023-11-08T08:18:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717400fbd0 con 0x7f7170003f80 2023-11-08T08:18:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.872+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71840cf390 con 0x7f7170003f80 2023-11-08T08:18:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.873+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71840743f0 con 0x7f7170003f80 2023-11-08T08:18:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.873+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f717400fd30 con 0x7f7170003f80 2023-11-08T08:18:53.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.874+0000 7f717a7fc700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71680682f0 0x7f716806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:53.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.874+0000 7f718956f700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71680682f0 0x7f716806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:53.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.874+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7174085df0 con 0x7f7170003f80 2023-11-08T08:18:53.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.874+0000 7f718956f700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71680682f0 0x7f716806a7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f71800090b0 tx=0x7f7180008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:53.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:53.878+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7174052650 con 0x7f7170003f80 2023-11-08T08:18:54.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.056+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f718406ad90 con 0x7f7170003f80 2023-11-08T08:18:54.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.057+0000 7f717a7fc700 1 -- 172.21.15.116:0/62178084 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7174055c70 con 0x7f7170003f80 2023-11-08T08:18:54.058 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:18:54.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71680682f0 msgr2=0x7f716806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71680682f0 0x7f716806a7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f71800090b0 tx=0x7f7180008030 comp rx=0 tx=0).stop 2023-11-08T08:18:54.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 msgr2=0x7f71840c6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f71840c6b90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7174007310 tx=0x7f7174007620 comp rx=0 tx=0).stop 2023-11-08T08:18:54.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 shutdown_connections 2023-11-08T08:18:54.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71841f1ff0 0x7f71841f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7170003f80 0x7f71840c6b90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71680682f0 0x7f716806a7a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 --2- 172.21.15.116:0/62178084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71840cbea0 0x7f71840c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 >> 172.21.15.116:0/62178084 conn(0x7f718406ea80 msgr2=0x7f71840cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:54.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 shutdown_connections 2023-11-08T08:18:54.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.059+0000 7f718bfd4700 1 -- 172.21.15.116:0/62178084 wait complete. 2023-11-08T08:18:54.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:18:54.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.190+0000 7fb7bf16e700 1 Processor -- start 2023-11-08T08:18:54.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.190+0000 7fb7bf16e700 1 -- start start 2023-11-08T08:18:54.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7bf16e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 0x7fb7b80b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7bf16e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b80b2110 con 0x7fb7b8142db0 2023-11-08T08:18:54.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7bf16e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b815e010 con 0x7fb7b80ce040 2023-11-08T08:18:54.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7bf16e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b813fc90 con 0x7fb7b80b1770 2023-11-08T08:18:54.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b7fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb7b8142db0 0x7fb7b813ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51676/0 (socket says 172.21.15.116:51676) 2023-11-08T08:18:54.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b7fff700 1 -- 172.21.15.116:0/3657891904 learned_addr learned my addr 172.21.15.116:0/3657891904 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:54.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7bd70b700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 0x7fb7b80b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275854680 0 0) 0x7fb7b80b2110 con 0x7fb7b8142db0 2023-11-08T08:18:54.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a80035b0 con 0x7fb7b8142db0 2023-11-08T08:18:54.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 258575712 0 0) 0x7fb7a80035b0 con 0x7fb7b8142db0 2023-11-08T08:18:54.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 msgr2=0x7fb7b80b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:54.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 0x7fb7b80b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 >> v1:172.21.15.129:6789/0 conn(0x7fb7b80ce040 legacy=0x7fb7b815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b813f480 con 0x7fb7b8142db0 2023-11-08T08:18:54.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fb7a4001670 con 0x7fb7b8142db0 2023-11-08T08:18:54.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 >> v1:172.21.15.116:6789/0 conn(0x7fb7b8142db0 legacy=0x7fb7b813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 0x7fb7a8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8003f70 0x7fb7a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a8004d90 0x7fb7a8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b813fc90 con 0x7fb7a8004d90 2023-11-08T08:18:54.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a800ba40 con 0x7fb7a8003f70 2023-11-08T08:18:54.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.191+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a800bc00 con 0x7fb7b80b1770 2023-11-08T08:18:54.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a8004d90 0x7fb7a8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7b7fff700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8003f70 0x7fb7a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bcf0a700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 0x7fb7a8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 -- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 msgr2=0x7fb7a8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:54.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b80b1770 0x7fb7a8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 -- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8003f70 msgr2=0x7fb7a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:54.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8003f70 0x7fb7a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 -- 172.21.15.116:0/3657891904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a80035b0 con 0x7fb7a8004d90 2023-11-08T08:18:54.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7bd70b700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a8004d90 0x7fb7a8005200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009fc0 tx=0x7fb7a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d65f7830d3ac43 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b000ec30 con 0x7fb7a8004d90 2023-11-08T08:18:54.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b000f040 con 0x7fb7a8004d90 2023-11-08T08:18:54.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.192+0000 7fb7b77fe700 1 -- 172.21.15.116:0/3657891904 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0014480 con 0x7fb7a8004d90 2023-11-08T08:18:54.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bf16e700 1 -- 172.21.15.116:0/3657891904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a8004d90 msgr2=0x7fb7a8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/3657891904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a8004d90 0x7fb7a8005200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009fc0 tx=0x7fb7a8005880 comp rx=0 tx=0).stop 2023-11-08T08:18:54.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bd70b700 1 -- 172.21.15.116:0/3657891904 reap_dead start 2023-11-08T08:18:54.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bf16e700 1 -- 172.21.15.116:0/3657891904 shutdown_connections 2023-11-08T08:18:54.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bf16e700 1 -- 172.21.15.116:0/3657891904 >> 172.21.15.116:0/3657891904 conn(0x7fb7b806c010 msgr2=0x7fb7b80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:54.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bf16e700 1 -- 172.21.15.116:0/3657891904 shutdown_connections 2023-11-08T08:18:54.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.193+0000 7fb7bf16e700 1 -- 172.21.15.116:0/3657891904 wait complete. 2023-11-08T08:18:54.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.194+0000 7fb7bf16e700 1 Processor -- start 2023-11-08T08:18:54.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.194+0000 7fb7bf16e700 1 -- start start 2023-11-08T08:18:54.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8004d90 0x7fb7b815a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 0x7fb7b80c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b8155a20 0x7fb7b80c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80b2ca0 con 0x7fb7b81555b0 2023-11-08T08:18:54.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80b2f60 con 0x7fb7a8004d90 2023-11-08T08:18:54.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80b3240 con 0x7fb7b8155a20 2023-11-08T08:18:54.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 0x7fb7b80c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 0x7fb7b80c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58742/0 (socket says 172.21.15.116:58742) 2023-11-08T08:18:54.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 -- 172.21.15.116:0/1308375214 learned_addr learned my addr 172.21.15.116:0/1308375214 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:54.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bcf0a700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8004d90 0x7fb7b815a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bd70b700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b8155a20 0x7fb7b80c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 -- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b8155a20 msgr2=0x7fb7b80c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b8155a20 0x7fb7b80c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 -- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8004d90 msgr2=0x7fb7b815a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8004d90 0x7fb7b815a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 -- 172.21.15.116:0/1308375214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b813f9b0 con 0x7fb7b81555b0 2023-11-08T08:18:54.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b7fff700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 0x7fb7b80c59f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007290 tx=0x7fb7a40075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a400f600 con 0x7fb7b81555b0 2023-11-08T08:18:54.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a400f7a0 con 0x7fb7b81555b0 2023-11-08T08:18:54.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b813f6d0 con 0x7fb7b81555b0 2023-11-08T08:18:54.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a400fb50 con 0x7fb7b81555b0 2023-11-08T08:18:54.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.195+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b815d100 con 0x7fb7b81555b0 2023-11-08T08:18:54.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.196+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b80743f0 con 0x7fb7b81555b0 2023-11-08T08:18:54.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.197+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7a400fcb0 con 0x7fb7b81555b0 2023-11-08T08:18:54.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.197+0000 7fb7b57fa700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb79c0682f0 0x7fb79c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.197+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb7a4085d20 con 0x7fb7b81555b0 2023-11-08T08:18:54.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.197+0000 7fb7bcf0a700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb79c0682f0 0x7fb79c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.197+0000 7fb7bcf0a700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb79c0682f0 0x7fb79c06a7a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac007fd0 tx=0x7fb7ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.201+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7a4052580 con 0x7fb7b81555b0 2023-11-08T08:18:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.424+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7b80c35a0 con 0x7fb7b81555b0 2023-11-08T08:18:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.424+0000 7fb7b57fa700 1 -- 172.21.15.116:0/1308375214 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4654 (secure 0 0 0) 0x7fb7a4055ba0 con 0x7fb7b81555b0 2023-11-08T08:18:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:18:54.426 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:18:42.885473+0000","modified":"2023-11-08T08:18:45.898738+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-08T08:18:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.426+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb79c0682f0 msgr2=0x7fb79c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.426+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb79c0682f0 0x7fb79c06a7a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac007fd0 tx=0x7fb7ac009040 comp rx=0 tx=0).stop 2023-11-08T08:18:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.426+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 msgr2=0x7fb7b80c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.426+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 0x7fb7b80c59f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007290 tx=0x7fb7a40075a0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 shutdown_connections 2023-11-08T08:18:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7b8155a20 0x7fb7b80c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb79c0682f0 0x7fb79c06a7a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7a8004d90 0x7fb7b815a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 --2- 172.21.15.116:0/1308375214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b81555b0 0x7fb7b80c59f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 >> 172.21.15.116:0/1308375214 conn(0x7fb7b806c010 msgr2=0x7fb7b806d620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 shutdown_connections 2023-11-08T08:18:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.427+0000 7fb7bf16e700 1 -- 172.21.15.116:0/1308375214 wait complete. 2023-11-08T08:18:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-08T08:18:54.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-08T08:18:42.885473+0000', 'modified': '2023-11-08T08:18:45.898738+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4305': {'gid': 4305, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/1536263974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 1536263974}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 1536263974}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-08T08:18:54.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-08T08:18:54.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-08T08:18:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.965+0000 7f4edecff700 1 Processor -- start 2023-11-08T08:18:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.966+0000 7f4edecff700 1 -- start start 2023-11-08T08:18:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.966+0000 7f4edecff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4ed80718e0 0x7f4ed806c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.966+0000 7f4edecff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed806cdc0 0x7f4ed806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.966+0000 7f4edecff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed8160a50 con 0x7f4ed815c810 2023-11-08T08:18:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.966+0000 7f4edecff700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed815cbe0 con 0x7f4ed80718e0 2023-11-08T08:18:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.966+0000 7f4edecff700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8160d30 con 0x7f4ed806cdc0 2023-11-08T08:18:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edd29c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed806cdc0 0x7f4ed806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4ed7fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4ed80718e0 0x7f4ed806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edca9b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4ed815c810 0x7f4ed8160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:53886/0 (socket says 172.21.15.116:53886) 2023-11-08T08:18:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edca9b700 1 -- 172.21.15.116:0/816460359 learned_addr learned my addr 172.21.15.116:0/816460359 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edd29c700 1 -- 172.21.15.116:0/816460359 >> v2:172.21.15.129:3300/0 conn(0x7f4ed80718e0 msgr2=0x7f4ed806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edd29c700 1 --2- 172.21.15.116:0/816460359 >> v2:172.21.15.129:3300/0 conn(0x7f4ed80718e0 0x7f4ed806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edd29c700 1 -- 172.21.15.116:0/816460359 >> v1:172.21.15.129:6789/0 conn(0x7f4ed815c810 legacy=0x7f4ed8160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edd29c700 1 -- 172.21.15.116:0/816460359 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8161010 con 0x7f4ed806cdc0 2023-11-08T08:18:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4ed77fe700 1 -- 172.21.15.116:0/816460359 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003158844 0 0) 0x7f4ed8160a50 con 0x7f4ed815c810 2023-11-08T08:18:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/816460359 >> v2:172.21.15.129:3300/0 conn(0x7f4ed80718e0 0x7f4ed806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4edd29c700 1 --2- 172.21.15.116:0/816460359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed806cdc0 0x7f4ed806a270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009dc0 tx=0x7f4ed000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fbc5bfa8f85705b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4ed77fe700 1 -- 172.21.15.116:0/816460359 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000e070 con 0x7f4ed806cdc0 2023-11-08T08:18:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4ed77fe700 1 -- 172.21.15.116:0/816460359 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed000bae0 con 0x7f4ed806cdc0 2023-11-08T08:18:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.967+0000 7f4ed77fe700 1 -- 172.21.15.116:0/816460359 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000be90 con 0x7f4ed806cdc0 2023-11-08T08:18:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 -- 172.21.15.116:0/816460359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed806cdc0 msgr2=0x7f4ed806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 --2- 172.21.15.116:0/816460359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed806cdc0 0x7f4ed806a270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009dc0 tx=0x7f4ed000b330 comp rx=0 tx=0).stop 2023-11-08T08:18:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 -- 172.21.15.116:0/816460359 shutdown_connections 2023-11-08T08:18:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 --2- 172.21.15.116:0/816460359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed806cdc0 0x7f4ed806a270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 --2- 172.21.15.116:0/816460359 >> v2:172.21.15.129:3300/0 conn(0x7f4ed80718e0 0x7f4ed806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 -- 172.21.15.116:0/816460359 >> 172.21.15.116:0/816460359 conn(0x7f4ed806c010 msgr2=0x7f4ed80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 -- 172.21.15.116:0/816460359 shutdown_connections 2023-11-08T08:18:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.968+0000 7f4edecff700 1 -- 172.21.15.116:0/816460359 wait complete. 2023-11-08T08:18:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.969+0000 7f4edecff700 1 Processor -- start 2023-11-08T08:18:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.969+0000 7f4edecff700 1 -- start start 2023-11-08T08:18:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edecff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed806cdc0 0x7f4ed815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edecff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed80718e0 0x7f4ed80b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edecff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 0x7f4ed81f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edecff700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed81576c0 con 0x7f4ed815c810 2023-11-08T08:18:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edecff700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed81567d0 con 0x7f4ed806cdc0 2023-11-08T08:18:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edecff700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8156ab0 con 0x7f4ed80718e0 2023-11-08T08:18:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 0x7f4ed81f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 0x7f4ed81f5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58754/0 (socket says 172.21.15.116:58754) 2023-11-08T08:18:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 -- 172.21.15.116:0/189054946 learned_addr learned my addr 172.21.15.116:0/189054946 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed80718e0 0x7f4ed80b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edca9b700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed806cdc0 0x7f4ed815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 -- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed80718e0 msgr2=0x7f4ed80b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed80718e0 0x7f4ed80b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 -- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed806cdc0 msgr2=0x7f4ed815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed806cdc0 0x7f4ed815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.970+0000 7f4edd29c700 1 -- 172.21.15.116:0/189054946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40094a0 con 0x7f4ed815c810 2023-11-08T08:18:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.971+0000 7f4edd29c700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 0x7f4ed81f5540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0014040 tx=0x7f4ed0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.971+0000 7f4ed57fa700 1 -- 172.21.15.116:0/189054946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0009910 con 0x7f4ed815c810 2023-11-08T08:18:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.971+0000 7f4ed57fa700 1 -- 172.21.15.116:0/189054946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0020070 con 0x7f4ed815c810 2023-11-08T08:18:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.971+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed0009a70 con 0x7f4ed815c810 2023-11-08T08:18:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.971+0000 7f4ed57fa700 1 -- 172.21.15.116:0/189054946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00276c0 con 0x7f4ed815c810 2023-11-08T08:18:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.971+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed80ce740 con 0x7f4ed815c810 2023-11-08T08:18:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.972+0000 7f4ed57fa700 1 -- 172.21.15.116:0/189054946 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ed002e070 con 0x7f4ed815c810 2023-11-08T08:18:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.972+0000 7f4ed57fa700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.972+0000 7f4ed57fa700 1 -- 172.21.15.116:0/189054946 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4ed0022070 con 0x7f4ed815c810 2023-11-08T08:18:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.972+0000 7f4edca9b700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.973+0000 7f4edca9b700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc000f60 tx=0x7f4ecc007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.974+0000 7f4eeae9eb80 1 Processor -- start 2023-11-08T08:18:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.974+0000 7f4eeae9eb80 1 -- start start 2023-11-08T08:18:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34aaca80 0x55eb34ab62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a70b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb349edde0 con 0x55eb34ab5f10 2023-11-08T08:18:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3495a140 con 0x55eb34aaca80 2023-11-08T08:18:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb349c8db0 con 0x55eb34a98620 2023-11-08T08:18:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58756/0 (socket says 172.21.15.116:58756) 2023-11-08T08:18:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 -- 172.21.15.116:0/275792426 learned_addr learned my addr 172.21.15.116:0/275792426 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34aaca80 0x55eb34ab62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edd29c700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a70b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 -- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 msgr2=0x55eb34a70b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a70b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 -- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34aaca80 msgr2=0x55eb34ab62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34aaca80 0x55eb34ab62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 -- 172.21.15.116:0/275792426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb347b6790 con 0x55eb34ab5f10 2023-11-08T08:18:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34aaca80 0x55eb34ab62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edd29c700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a70b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.975+0000 7f4edca9b700 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc015810 tx=0x7f4ecc004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6272f2efc78c74f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4ec27fc700 1 -- 172.21.15.116:0/275792426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc015110 con 0x55eb34ab5f10 2023-11-08T08:18:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4ec27fc700 1 -- 172.21.15.116:0/275792426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc015270 con 0x55eb34ab5f10 2023-11-08T08:18:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4ec27fc700 1 -- 172.21.15.116:0/275792426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc021650 con 0x55eb34ab5f10 2023-11-08T08:18:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/275792426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 msgr2=0x55eb34a6f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc015810 tx=0x7f4ecc004f90 comp rx=0 tx=0).stop 2023-11-08T08:18:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/275792426 shutdown_connections 2023-11-08T08:18:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a70b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/275792426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34aaca80 0x55eb34ab62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/275792426 >> 172.21.15.116:0/275792426 conn(0x55eb34a65ce0 msgr2=0x55eb34a6eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/275792426 shutdown_connections 2023-11-08T08:18:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.976+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/275792426 wait complete. 2023-11-08T08:18:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.977+0000 7f4eeae9eb80 1 -- start start 2023-11-08T08:18:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.977+0000 7f4eeae9eb80 1 Processor -- start 2023-11-08T08:18:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34a98620 0x55eb34aaec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb349c2120 con 0x55eb34aaca80 2023-11-08T08:18:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb347b7390 con 0x55eb34a98620 2023-11-08T08:18:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb3495a340 con 0x55eb34ab5f10 2023-11-08T08:18:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 10 client.0 _ll_register_callbacks cb 0x55eb34960400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:18:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58758/0 (socket says 172.21.15.116:58758) 2023-11-08T08:18:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 -- 172.21.15.116:0/801114334 learned_addr learned my addr 172.21.15.116:0/801114334 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4edd29c700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4edca9b700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34a98620 0x55eb34aaec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 -- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34ab5f10 msgr2=0x55eb34ab3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 -- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34a98620 msgr2=0x55eb34aaec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34a98620 0x55eb34aaec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 -- 172.21.15.116:0/801114334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc015550 con 0x55eb34aaca80 2023-11-08T08:18:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40149c0 tx=0x7f4ec4014cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b41bec1859e860b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eb77fe700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:18:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eb77fe700 1 -- 172.21.15.116:0/801114334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4004030 con 0x55eb34aaca80 2023-11-08T08:18:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eb77fe700 1 -- 172.21.15.116:0/801114334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4ec401a070 con 0x55eb34aaca80 2023-11-08T08:18:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.978+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb349c6030 con 0x55eb34aaca80 2023-11-08T08:18:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eb77fe700 1 -- 172.21.15.116:0/801114334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ec4004190 con 0x55eb34aaca80 2023-11-08T08:18:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 10 client.4644 fetch_fsmap learned FSMap version 8 2023-11-08T08:18:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55eb348a30a0 con 0x55eb34aaca80 2023-11-08T08:18:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eb77fe700 1 -- 172.21.15.116:0/801114334 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55eb349c6030 con 0x55eb34aaca80 2023-11-08T08:18:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eb77fe700 1 -- 172.21.15.116:0/801114334 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 8) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f4ec4011070 con 0x55eb34aaca80 2023-11-08T08:18:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 10 client.4644 fetch_fsmap finished waiting for FSMap version 8 2023-11-08T08:18:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 10 client.4644 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-08T08:18:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 20 client.4644 populate_metadata read hostname 'smithi116' 2023-11-08T08:18:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34aad5f0 0x55eb34aad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 4 client.4644 mds_command: new command op to 4305 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:18:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 --> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb34811ef0 con 0x55eb34aad5f0 2023-11-08T08:18:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4edca9b700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34aad5f0 0x55eb34aad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4edca9b700 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34aad5f0 0x55eb34aad9c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e6f2ac2af0c796 server_cookie=a9eb56390b28d2f6 in_seq=0 out_seq=0 2023-11-08T08:18:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.979+0000 7f4eb77fe700 10 client.4644 ms_handle_connect on v2:172.21.15.129:6834/1536263974 2023-11-08T08:18:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eb77fe700 1 -- 172.21.15.116:0/801114334 <== mds.0 v2:172.21.15.129:6834/1536263974 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55eb34811ef0 con 0x55eb34aad5f0 2023-11-08T08:18:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eb77fe700 10 client.4644 handle_command_reply: tid=0 2023-11-08T08:18:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 client.4644 shutdown 2023-11-08T08:18:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 msgr2=0x55eb34ab3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40149c0 tx=0x7f4ec4014cd0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 shutdown_connections 2023-11-08T08:18:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34a98620 0x55eb34aaec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34aad5f0 0x55eb34aad9c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/801114334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eb77fe700 0 client.4644 ms_handle_reset on v2:172.21.15.129:6834/1536263974 2023-11-08T08:18:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 >> 172.21.15.116:0/801114334 conn(0x55eb34a65ce0 msgr2=0x55eb34a6b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 shutdown_connections 2023-11-08T08:18:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.980+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/801114334 wait complete. 2023-11-08T08:18:55.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.981+0000 7f4eeae9eb80 20 client.4644 trim_cache size 0 max 16384 2023-11-08T08:18:55.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.998+0000 7f4eeae9eb80 1 Processor -- start 2023-11-08T08:18:55.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.998+0000 7f4eeae9eb80 1 -- start start 2023-11-08T08:18:55.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.998+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 0x55eb34ab3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34aaca80 0x55eb34ab3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb34a70eb0 con 0x55eb34a98620 2023-11-08T08:18:55.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb34a9c1c0 con 0x55eb34ab5f10 2023-11-08T08:18:55.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb34a6c7a0 con 0x55eb34aaca80 2023-11-08T08:18:55.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 0x55eb34ab3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 0x55eb34ab3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58774/0 (socket says 172.21.15.116:58774) 2023-11-08T08:18:55.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 -- 172.21.15.116:0/1886897145 learned_addr learned my addr 172.21.15.116:0/1886897145 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:55.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 -- 172.21.15.116:0/1886897145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34aaca80 msgr2=0x55eb34ab3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34aaca80 0x55eb34ab3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 -- 172.21.15.116:0/1886897145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 msgr2=0x55eb34a6f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 -- 172.21.15.116:0/1886897145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed81f2f60 con 0x55eb34a98620 2023-11-08T08:18:55.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4edca9b700 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 0x55eb34ab3a20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc004980 tx=0x7f4ecc00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160360e5978fcb12 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:55.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4ec2ffd700 1 -- 172.21.15.116:0/1886897145 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc027070 con 0x55eb34a98620 2023-11-08T08:18:55.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4ec2ffd700 1 -- 172.21.15.116:0/1886897145 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc019040 con 0x55eb34a98620 2023-11-08T08:18:55.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:54.999+0000 7f4ec2ffd700 1 -- 172.21.15.116:0/1886897145 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc02ea40 con 0x55eb34a98620 2023-11-08T08:18:55.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/1886897145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 msgr2=0x55eb34ab3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 0x55eb34ab3a20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc004980 tx=0x7f4ecc00a000 comp rx=0 tx=0).stop 2023-11-08T08:18:55.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/1886897145 shutdown_connections 2023-11-08T08:18:55.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34a6f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34a98620 0x55eb34ab3a20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/1886897145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34aaca80 0x55eb34ab3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/1886897145 >> 172.21.15.116:0/1886897145 conn(0x55eb34a65ce0 msgr2=0x55eb34ab2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/1886897145 shutdown_connections 2023-11-08T08:18:55.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/1886897145 wait complete. 2023-11-08T08:18:55.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 -- start start 2023-11-08T08:18:55.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.000+0000 7f4eeae9eb80 1 Processor -- start 2023-11-08T08:18:55.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a9ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb34b1c3b0 con 0x55eb34aaca80 2023-11-08T08:18:55.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb34a96560 con 0x55eb34ab5f10 2023-11-08T08:18:55.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb34a9dfd0 con 0x55eb34a98620 2023-11-08T08:18:55.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 10 client.0 _ll_register_callbacks cb 0x55eb34960400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:18:55.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58780/0 (socket says 172.21.15.116:58780) 2023-11-08T08:18:55.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 -- 172.21.15.116:0/3571225357 learned_addr learned my addr 172.21.15.116:0/3571225357 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:55.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4edd29c700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4edca9b700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 -- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 msgr2=0x55eb34a9ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a9ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 -- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 msgr2=0x55eb34ab3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 -- 172.21.15.116:0/3571225357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb347b6790 con 0x55eb34aaca80 2023-11-08T08:18:55.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4ed7fff700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4014d10 tx=0x7f4ec40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5212ee836d5b72 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:55.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eb77fe700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:18:55.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eb77fe700 1 -- 172.21.15.116:0/3571225357 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4012830 con 0x55eb34aaca80 2023-11-08T08:18:55.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eb77fe700 1 -- 172.21.15.116:0/3571225357 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4ec4000e40 con 0x55eb34aaca80 2023-11-08T08:18:55.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ecc015270 con 0x55eb34aaca80 2023-11-08T08:18:55.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eb77fe700 1 -- 172.21.15.116:0/3571225357 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ec4012990 con 0x55eb34aaca80 2023-11-08T08:18:55.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 10 client.4650 fetch_fsmap learned FSMap version 8 2023-11-08T08:18:55.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55eb3496fb10 con 0x55eb34aaca80 2023-11-08T08:18:55.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.002+0000 7f4eb77fe700 1 -- 172.21.15.116:0/3571225357 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec4038020 con 0x55eb34aaca80 2023-11-08T08:18:55.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eb77fe700 1 -- 172.21.15.116:0/3571225357 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 8) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f4ec401f480 con 0x55eb34aaca80 2023-11-08T08:18:55.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2023-11-08T08:18:55.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 10 client.4650 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-08T08:18:55.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 20 client.4650 populate_metadata read hostname 'smithi116' 2023-11-08T08:18:55.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34b1b880 0x55eb34a90690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 4 client.4650 mds_command: new command op to 4305 tid=0[{"prefix": "status"}] 2023-11-08T08:18:55.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 --> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eb34847040 con 0x55eb34b1b880 2023-11-08T08:18:55.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4edca9b700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34b1b880 0x55eb34a90690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eb77fe700 10 client.4650 ms_handle_connect on v2:172.21.15.129:6834/1536263974 2023-11-08T08:18:55.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4edca9b700 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34b1b880 0x55eb34a90690 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55d7aaae2cafbf0 server_cookie=2c6c6a4bd1bc4ed6 in_seq=0 out_seq=0 2023-11-08T08:18:55.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eb77fe700 1 -- 172.21.15.116:0/3571225357 <== mds.0 v2:172.21.15.129:6834/1536263974 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55eb34847040 con 0x55eb34b1b880 2023-11-08T08:18:55.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eb77fe700 10 client.4650 handle_command_reply: tid=0 2023-11-08T08:18:55.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 client.4650 shutdown 2023-11-08T08:18:55.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 msgr2=0x55eb34ab3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4014d10 tx=0x7f4ec40047c0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 shutdown_connections 2023-11-08T08:18:55.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb34ab5f10 0x55eb34ab3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55eb34a98620 0x55eb34a9ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.129:6834/1536263974,v1:172.21.15.129:6835/1536263974] conn(0x55eb34b1b880 0x55eb34a90690 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eeae9eb80 1 --2- 172.21.15.116:0/3571225357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb34aaca80 0x55eb34ab3a20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.003+0000 7f4eb77fe700 0 client.4650 ms_handle_reset on v2:172.21.15.129:6834/1536263974 2023-11-08T08:18:55.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 >> 172.21.15.116:0/3571225357 conn(0x55eb34a65ce0 msgr2=0x55eb34aaf350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 shutdown_connections 2023-11-08T08:18:55.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4eeae9eb80 1 -- 172.21.15.116:0/3571225357 wait complete. 2023-11-08T08:18:55.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4eeae9eb80 20 client.4650 trim_cache size 0 max 16384 2023-11-08T08:18:55.035 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:18:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "2f470591-d661-4b50-b7a5-062dfcf46b1d", 2023-11-08T08:18:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-08T08:18:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4305, 2023-11-08T08:18:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-08T08:18:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-08T08:18:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-08T08:18:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 12.117551141, 2023-11-08T08:18:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 8, 2023-11-08T08:18:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 20, 2023-11-08T08:18:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 20, 2023-11-08T08:18:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 17.411857001000001 2023-11-08T08:18:55.037 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:18:55.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ebc0682f0 msgr2=0x7f4ebc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc000f60 tx=0x7f4ecc007e70 comp rx=0 tx=0).stop 2023-11-08T08:18:55.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 msgr2=0x7f4ed81f5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 0x7f4ed81f5540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0014040 tx=0x7f4ed0004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 shutdown_connections 2023-11-08T08:18:55.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ed815c810 0x7f4ed81f5540 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed806cdc0 0x7f4ed815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 --2- 172.21.15.116:0/189054946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ed80718e0 0x7f4ed80b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 >> 172.21.15.116:0/189054946 conn(0x7f4ed806c010 msgr2=0x7f4ed80cc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 shutdown_connections 2023-11-08T08:18:55.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.004+0000 7f4edecff700 1 -- 172.21.15.116:0/189054946 wait complete. 2023-11-08T08:18:55.041 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-11-08T08:18:55.064 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-11-08T08:18:55.064 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-08T08:18:55.065 INFO:tasks.ceph_fuse:client.0 config is {} 2023-11-08T08:18:55.065 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-08T08:18:55.066 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-08T08:18:55.066 INFO:tasks.ceph_fuse:client.2 config is {} 2023-11-08T08:18:55.066 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-08T08:18:55.067 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-08T08:18:55.067 INFO:tasks.ceph_fuse:client.1 config is {} 2023-11-08T08:18:55.067 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-08T08:18:55.067 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-08T08:18:55.068 INFO:tasks.ceph_fuse:client.3 config is {} 2023-11-08T08:18:55.068 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-08T08:18:55.068 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-08T08:18:55.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.069 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:18:55.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2023-11-08T08:18:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2023-11-08T08:18:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:18:55.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.159 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2023-11-08T08:18:55.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2023-11-08T08:18:55.257 INFO:teuthology.orchestra.run.smithi129.stderr:Cannot find device "ceph-brx" 2023-11-08T08:18:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:18:55.259 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-11-08T08:18:55.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:18:55.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.379+0000 7f13a6ba7700 1 Processor -- start 2023-11-08T08:18:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.379+0000 7f13a6ba7700 1 -- start start 2023-11-08T08:18:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a6ba7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f13a006ce20 0x7f13a015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a6ba7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a015db00 0x7f13a006d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a6ba7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13a00c9230 0x7f13a00ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a6ba7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a015a060 con 0x7f13a00c9230 2023-11-08T08:18:55.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a6ba7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a0159a80 con 0x7f13a006ce20 2023-11-08T08:18:55.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a6ba7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a006d730 con 0x7f13a015db00 2023-11-08T08:18:55.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13a00c9230 0x7f13a00ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13a00c9230 0x7f13a00ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58792/0 (socket says 172.21.15.116:58792) 2023-11-08T08:18:55.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 -- 172.21.15.116:0/3178233261 learned_addr learned my addr 172.21.15.116:0/3178233261 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:55.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a4943700 1 --2- 172.21.15.116:0/3178233261 >> v2:172.21.15.129:3300/0 conn(0x7f13a006ce20 0x7f13a015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f139ffff700 1 --2- 172.21.15.116:0/3178233261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a015db00 0x7f13a006d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 -- 172.21.15.116:0/3178233261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a015db00 msgr2=0x7f13a006d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 --2- 172.21.15.116:0/3178233261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a015db00 0x7f13a006d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 -- 172.21.15.116:0/3178233261 >> v2:172.21.15.129:3300/0 conn(0x7f13a006ce20 msgr2=0x7f13a015a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:55.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 --2- 172.21.15.116:0/3178233261 >> v2:172.21.15.129:3300/0 conn(0x7f13a006ce20 0x7f13a015a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 -- 172.21.15.116:0/3178233261 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00c9ac0 con 0x7f13a00c9230 2023-11-08T08:18:55.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f13a5144700 1 --2- 172.21.15.116:0/3178233261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a00c9230 0x7f13a00ccc80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f13a006fa10 tx=0x7f139800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848c8bd23c2f3435 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:55.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f139f7fe700 1 -- 172.21.15.116:0/3178233261 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398010070 con 0x7f13a00c9230 2023-11-08T08:18:55.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f139f7fe700 1 -- 172.21.15.116:0/3178233261 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139800dd90 con 0x7f13a00c9230 2023-11-08T08:18:55.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.380+0000 7f139f7fe700 1 -- 172.21.15.116:0/3178233261 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139800b310 con 0x7f13a00c9230 2023-11-08T08:18:55.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 -- 172.21.15.116:0/3178233261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a00c9230 msgr2=0x7f13a00ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/3178233261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a00c9230 0x7f13a00ccc80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f13a006fa10 tx=0x7f139800d430 comp rx=0 tx=0).stop 2023-11-08T08:18:55.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 -- 172.21.15.116:0/3178233261 shutdown_connections 2023-11-08T08:18:55.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/3178233261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a00c9230 0x7f13a00ccc80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/3178233261 >> v2:172.21.15.129:3300/0 conn(0x7f13a006ce20 0x7f13a015a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/3178233261 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a015db00 0x7f13a006d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 -- 172.21.15.116:0/3178233261 >> 172.21.15.116:0/3178233261 conn(0x7f13a00cb660 msgr2=0x7f13a00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 -- 172.21.15.116:0/3178233261 shutdown_connections 2023-11-08T08:18:55.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.381+0000 7f13a6ba7700 1 -- 172.21.15.116:0/3178233261 wait complete. 2023-11-08T08:18:55.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.382+0000 7f13a6ba7700 1 Processor -- start 2023-11-08T08:18:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.383+0000 7f13a6ba7700 1 -- start start 2023-11-08T08:18:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a006ce20 0x7f13a0155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a00c9230 0x7f13a0158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 0x7f13a0143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0156270 con 0x7f13a015db00 2023-11-08T08:18:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0157700 con 0x7f13a00c9230 2023-11-08T08:18:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a01579a0 con 0x7f13a006ce20 2023-11-08T08:18:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 0x7f13a0143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 0x7f13a0143be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58808/0 (socket says 172.21.15.116:58808) 2023-11-08T08:18:55.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 -- 172.21.15.116:0/1455275561 learned_addr learned my addr 172.21.15.116:0/1455275561 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:55.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f139ffff700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a00c9230 0x7f13a0158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a4943700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a006ce20 0x7f13a0155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 -- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a006ce20 msgr2=0x7f13a0155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a006ce20 0x7f13a0155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 -- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a00c9230 msgr2=0x7f13a0158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a00c9230 0x7f13a0158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 -- 172.21.15.116:0/1455275561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139800bd40 con 0x7f13a015db00 2023-11-08T08:18:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f139ffff700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a00c9230 0x7f13a0158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:18:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a5144700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 0x7f13a0143be0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13a00b17d0 tx=0x7f1398004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398009810 con 0x7f13a015db00 2023-11-08T08:18:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13980099b0 con 0x7f13a015db00 2023-11-08T08:18:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a0159790 con 0x7f13a015db00 2023-11-08T08:18:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398009d60 con 0x7f13a015db00 2023-11-08T08:18:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.384+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a0159cf0 con 0x7f13a015db00 2023-11-08T08:18:55.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.385+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a00c9da0 con 0x7f13a015db00 2023-11-08T08:18:55.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.386+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1398004040 con 0x7f13a015db00 2023-11-08T08:18:55.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.386+0000 7f139d7fa700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13840682f0 0x7f138406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:55.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.386+0000 7f13a4943700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13840682f0 0x7f138406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:55.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.386+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f13980895b0 con 0x7f13a015db00 2023-11-08T08:18:55.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.386+0000 7f13a4943700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13840682f0 0x7f138406a7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f13940096f0 tx=0x7f1394008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:55.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.390+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1398055db0 con 0x7f13a015db00 2023-11-08T08:18:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.616+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f13a00743f0 con 0x7f13a015db00 2023-11-08T08:18:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.616+0000 7f139d7fa700 1 -- 172.21.15.116:0/1455275561 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1398022060 con 0x7f13a015db00 2023-11-08T08:18:55.618 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:18:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13840682f0 msgr2=0x7f138406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13840682f0 0x7f138406a7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f13940096f0 tx=0x7f1394008040 comp rx=0 tx=0).stop 2023-11-08T08:18:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 msgr2=0x7f13a0143be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 0x7f13a0143be0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13a00b17d0 tx=0x7f1398004ad0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 shutdown_connections 2023-11-08T08:18:55.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a015db00 0x7f13a0143be0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a006ce20 0x7f13a0155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13840682f0 0x7f138406a7a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 --2- 172.21.15.116:0/1455275561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a00c9230 0x7f13a0158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:55.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 >> 172.21.15.116:0/1455275561 conn(0x7f13a00cb660 msgr2=0x7f13a00b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 shutdown_connections 2023-11-08T08:18:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:55.618+0000 7f13a6ba7700 1 -- 172.21.15.116:0/1455275561 wait complete. 2023-11-08T08:18:55.630 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:18:55.631 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:18:55.631 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-08T08:18:55.631 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-08T08:18:55.632 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-08T08:18:55.632 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:18:55.632 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-08T08:18:55.632 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:18:55.633 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:18:55.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.633 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-08T08:18:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:18:55.651 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:18:55.651 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:18:55.651 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:55.652 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:18:55.652 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:55.652 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:18:55.652 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:55.653 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-08T08:18:55.653 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:18:55.653 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:55.653 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-08T08:18:55.653 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:18:55.654 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:55.654 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:18:55.654 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:55.654 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:18:55.654 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42282sec preferred_lft 42282sec 2023-11-08T08:18:55.655 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-08T08:18:55.655 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:55.655 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-08T08:18:55.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.656 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:55.656 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:55.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2023-11-08T08:18:55.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2023-11-08T08:18:55.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2023-11-08T08:18:55.658 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-08T08:18:55.658 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:55.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.834 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:18:55.862 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-08T08:18:55.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.863 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-08T08:18:55.942 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-08T08:18:55.943 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:18:55.943 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:18:55.943 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:18:55.943 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:18:55.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:55.944 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:55.945 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:55.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:18:55.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:18:55.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:18:55.946 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:56.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:56.221 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:18:56.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:56.240 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2023-11-08T08:18:56.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:56.297 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:56.297 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:56.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-08T08:18:56.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-08T08:18:56.298 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:56.391 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-08T08:18:56.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:56.392 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:56.392 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:56.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-08T08:18:56.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-08T08:18:56.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-08T08:18:56.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-08T08:18:56.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-08T08:18:56.394 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:56.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:56.591 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:56.591 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:56.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-08T08:18:56.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2023-11-08T08:18:56.592 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:56.688 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-08T08:18:56.689 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:18:56.689 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-08T08:18:56.707 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:18:56.707 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:18:56.708 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:18:56.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:18:56.831 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:18:56.834 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:18:56.897 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:18:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:18:56.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:56.899 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:18:56.956 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-08T08:18:56.957 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-11-08T08:18:56.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:18:57.024 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:18:57.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:18:57.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-08T08:18:57.084+0000 7f1e7a8ab5c0 -1 init, newargv = 0x564ddaa38a30 newargc=15 2023-11-08T08:18:57.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[130851]: starting ceph client 2023-11-08T08:18:57.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[130851]: starting fuse 2023-11-08T08:18:57.092 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:18:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:18:57.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:57.097 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:18:57.154 INFO:teuthology.orchestra.run.smithi116.stdout:46 2023-11-08T08:18:57.154 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46] 2023-11-08T08:18:57.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:18:57.155 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-08T08:18:57.155 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-08T08:18:57.156 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:18:57.156 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-08T08:18:57.156 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:57.157 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-08T08:18:57.157 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-08T08:18:57.157 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-08T08:18:57.158 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-08T08:18:57.158 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:57.158 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-08T08:18:57.159 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-08T08:18:57.159 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-08T08:18:57.159 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-08T08:18:57.160 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:57.160 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-08T08:18:57.160 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:18:57.161 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:18:57.161 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:18:57.161 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-08T08:18:57.161 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-08T08:18:57.162 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-08T08:18:57.162 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:18:57.162 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:57.163 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-08T08:18:57.163 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:18:57.276 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.130851.asok 2023-11-08T08:18:57.459 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.130851.asok 2023-11-08T08:18:57.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:57.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.130851.asok status 2023-11-08T08:18:57.615 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:18:57.615 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-08T08:18:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:18:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:18:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-08T08:18:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-08T08:18:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-08T08:18:57.617 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "130851", 2023-11-08T08:18:57.617 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-08T08:18:57.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:57.617 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-08T08:18:57.617 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-08T08:18:57.618 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4662, 2023-11-08T08:18:57.618 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-08T08:18:57.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-08T08:18:57.618 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-08T08:18:57.618 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4662 2023-11-08T08:18:57.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:57.619 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:18:57.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:18:57.619 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:18:57.619 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2695970067 2023-11-08T08:18:57.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:18:57.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:57.620 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:18:57.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:18:57.620 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:18:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2695970067 2023-11-08T08:18:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.4662 192.168.0.1:0/2695970067", 2023-11-08T08:18:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/2695970067", 2023-11-08T08:18:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-08T08:18:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 8, 2023-11-08T08:18:57.622 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 20, 2023-11-08T08:18:57.622 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:18:57.622 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-08T08:18:57.622 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-08T08:18:57.622 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:18:57.625 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:18:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.744+0000 7f39fb92f700 1 Processor -- start 2023-11-08T08:18:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.745+0000 7f39fb92f700 1 -- start start 2023-11-08T08:18:57.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.745+0000 7f39fb92f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 0x7f39f4071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:57.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.745+0000 7f39fb92f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f39f40ae520 0x7f39f406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:57.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.745+0000 7f39fb92f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f40af0c0 con 0x7f39f40b43f0 2023-11-08T08:18:57.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.745+0000 7f39fb92f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f40b0590 con 0x7f39f40ae520 2023-11-08T08:18:57.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.745+0000 7f39fb92f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f4159280 con 0x7f39f406a270 2023-11-08T08:18:57.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 0x7f39f4071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f8eca700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f39f40ae520 0x7f39f406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f9ecc700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f39f40b43f0 0x7f39f40b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:53892/0 (socket says 172.21.15.116:53892) 2023-11-08T08:18:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f9ecc700 1 -- 172.21.15.116:0/2144970105 learned_addr learned my addr 172.21.15.116:0/2144970105 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 0x7f39f4071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:41960/0 (socket says 172.21.15.116:41960) 2023-11-08T08:18:57.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 -- 172.21.15.116:0/2144970105 >> v2:172.21.15.129:3300/0 conn(0x7f39f40ae520 msgr2=0x7f39f406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:57.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 --2- 172.21.15.116:0/2144970105 >> v2:172.21.15.129:3300/0 conn(0x7f39f40ae520 0x7f39f406a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 -- 172.21.15.116:0/2144970105 >> v1:172.21.15.129:6789/0 conn(0x7f39f40b43f0 legacy=0x7f39f40b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:57.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 -- 172.21.15.116:0/2144970105 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f406ab80 con 0x7f39f406a270 2023-11-08T08:18:57.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f8eca700 1 --2- 172.21.15.116:0/2144970105 >> v2:172.21.15.129:3300/0 conn(0x7f39f40ae520 0x7f39f406a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:18:57.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39f96cb700 1 --2- 172.21.15.116:0/2144970105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 0x7f39f4071f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f39e400a9e0 tx=0x7f39e400acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4db1d97ef7b63201 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.746+0000 7f39ebfff700 1 -- 172.21.15.116:0/2144970105 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400cb70 con 0x7f39f406a270 2023-11-08T08:18:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39ebfff700 1 -- 172.21.15.116:0/2144970105 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e400ccd0 con 0x7f39f406a270 2023-11-08T08:18:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39ebfff700 1 -- 172.21.15.116:0/2144970105 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4018620 con 0x7f39f406a270 2023-11-08T08:18:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 -- 172.21.15.116:0/2144970105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 msgr2=0x7f39f4071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 --2- 172.21.15.116:0/2144970105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 0x7f39f4071f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f39e400a9e0 tx=0x7f39e400acf0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 -- 172.21.15.116:0/2144970105 shutdown_connections 2023-11-08T08:18:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 --2- 172.21.15.116:0/2144970105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f406a270 0x7f39f4071f70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 --2- 172.21.15.116:0/2144970105 >> v2:172.21.15.129:3300/0 conn(0x7f39f40ae520 0x7f39f406a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 -- 172.21.15.116:0/2144970105 >> 172.21.15.116:0/2144970105 conn(0x7f39f40b2fc0 msgr2=0x7f39f4071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:57.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 -- 172.21.15.116:0/2144970105 shutdown_connections 2023-11-08T08:18:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.747+0000 7f39fb92f700 1 -- 172.21.15.116:0/2144970105 wait complete. 2023-11-08T08:18:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.748+0000 7f39fb92f700 1 Processor -- start 2023-11-08T08:18:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.748+0000 7f39fb92f700 1 -- start start 2023-11-08T08:18:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39fb92f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f406a270 0x7f39f41f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39fb92f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f40ae520 0x7f39f4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39fb92f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 0x7f39f40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39fb92f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f415a950 con 0x7f39f40b43f0 2023-11-08T08:18:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39fb92f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f415ac30 con 0x7f39f406a270 2023-11-08T08:18:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39fb92f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f415af10 con 0x7f39f40ae520 2023-11-08T08:18:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 0x7f39f40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 0x7f39f40b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58816/0 (socket says 172.21.15.116:58816) 2023-11-08T08:18:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 -- 172.21.15.116:0/1666508753 learned_addr learned my addr 172.21.15.116:0/1666508753 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f96cb700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f406a270 0x7f39f41f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f8eca700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f40ae520 0x7f39f4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 -- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f40ae520 msgr2=0x7f39f4159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f40ae520 0x7f39f4159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 -- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f406a270 msgr2=0x7f39f41f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.749+0000 7f39f9ecc700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f406a270 0x7f39f41f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39f9ecc700 1 -- 172.21.15.116:0/1666508753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00094a0 con 0x7f39f40b43f0 2023-11-08T08:18:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39f9ecc700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 0x7f39f40b47f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00cbc0 tx=0x7f39ec00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec00a9e0 con 0x7f39f40b43f0 2023-11-08T08:18:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec00ab40 con 0x7f39f40b43f0 2023-11-08T08:18:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec012400 con 0x7f39f40b43f0 2023-11-08T08:18:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e400a640 con 0x7f39f40b43f0 2023-11-08T08:18:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.750+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f4072670 con 0x7f39f40b43f0 2023-11-08T08:18:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.751+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f40b0d10 con 0x7f39f40b43f0 2023-11-08T08:18:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.751+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f39ec019030 con 0x7f39f40b43f0 2023-11-08T08:18:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.752+0000 7f39e9ffb700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f39d80682f0 0x7f39d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.752+0000 7f39f96cb700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f39d80682f0 0x7f39d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.752+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f39ec085620 con 0x7f39f40b43f0 2023-11-08T08:18:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.752+0000 7f39f96cb700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f39d80682f0 0x7f39d806a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f39e4000c00 tx=0x7f39e4012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.756+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39ec04b5e0 con 0x7f39f40b43f0 2023-11-08T08:18:57.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.978+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f39f40997f0 con 0x7f39f40b43f0 2023-11-08T08:18:57.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.978+0000 7f39e9ffb700 1 -- 172.21.15.116:0/1666508753 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f39f40997f0 con 0x7f39f40b43f0 2023-11-08T08:18:57.980 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:18:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.980+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f39d80682f0 msgr2=0x7f39d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.980+0000 7f39fb92f700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f39d80682f0 0x7f39d806a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f39e4000c00 tx=0x7f39e4012000 comp rx=0 tx=0).stop 2023-11-08T08:18:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.980+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 msgr2=0x7f39f40b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.980+0000 7f39fb92f700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 0x7f39f40b47f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00cbc0 tx=0x7f39ec00ced0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 shutdown_connections 2023-11-08T08:18:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39f40b43f0 0x7f39f40b47f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f39d80682f0 0x7f39d806a7a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f406a270 0x7f39f41f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 --2- 172.21.15.116:0/1666508753 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f39f40ae520 0x7f39f4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:57.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 >> 172.21.15.116:0/1666508753 conn(0x7f39f40b2fc0 msgr2=0x7f39f406d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:57.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 shutdown_connections 2023-11-08T08:18:57.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:57.981+0000 7f39fb92f700 1 -- 172.21.15.116:0/1666508753 wait complete. 2023-11-08T08:18:57.993 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:18:57.994 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:18:57.994 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-08T08:18:57.994 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-08T08:18:57.994 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-08T08:18:57.995 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:18:57.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-08T08:18:57.995 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:18:57.996 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-08T08:18:57.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:57.996 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-08T08:18:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:18:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:18:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:18:58.015 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:18:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.016 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:18:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:58.016 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-08T08:18:58.017 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:18:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-08T08:18:58.017 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:18:58.017 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:58.017 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:18:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:18:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42279sec preferred_lft 42279sec 2023-11-08T08:18:58.018 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-08T08:18:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.019 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:18:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:18:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-08T08:18:58.019 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::8c0:d4ff:fe3b:7f6b/64 scope link 2023-11-08T08:18:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.020 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:18:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-08T08:18:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bc76:a3ff:fe22:41c7/64 scope link tentative 2023-11-08T08:18:58.021 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.021 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:18:58.080 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-08T08:18:58.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.082 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-08T08:18:58.142 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-08T08:18:58.142 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:18:58.142 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:18:58.143 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:18:58.143 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:18:58.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.144 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:58.144 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:58.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:18:58.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:18:58.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:18:58.145 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:58.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.245 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:18:58.263 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-08T08:18:58.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.264 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2023-11-08T08:18:58.321 INFO:teuthology.orchestra.run.smithi116.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-08T08:18:58.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.322 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:58.322 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:58.322 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-08T08:18:58.323 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-08T08:18:58.323 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:58.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-08T08:18:58.448 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:18:58.448 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:18:58.448 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:18:58.448 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.449 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:18:58.449 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.449 INFO:teuthology.orchestra.run.smithi116.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:18:58.449 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 6e:64:3c:0f:08:7f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-08T08:18:58.450 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-08T08:18:58.450 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.450 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::6c64:3cff:fe0f:87f/64 scope link 2023-11-08T08:18:58.450 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.467 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-08T08:18:58.546 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:18:58.546 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:18:58.546 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:18:58.547 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.547 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:18:58.547 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.547 INFO:teuthology.orchestra.run.smithi116.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:18:58.548 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 6e:64:3c:0f:08:7f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-08T08:18:58.548 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-08T08:18:58.548 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.548 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::6c64:3cff:fe0f:87f/64 scope link 2023-11-08T08:18:58.548 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:18:58.564 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-08T08:18:58.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.565 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:58.565 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:58.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-08T08:18:58.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-08T08:18:58.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-08T08:18:58.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-08T08:18:58.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-08T08:18:58.567 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:58.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:58.794 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:18:58.794 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:18:58.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.1 up 2023-11-08T08:18:58.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.1 master ceph-brx 2023-11-08T08:18:58.795 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:18:58.891 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-11-08T08:18:58.892 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:18:58.892 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-08T08:18:58.909 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-08T08:18:58.910 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:18:58.910 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-08T08:18:58.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:18:59.034 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:18:59.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:18:59.103 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:18:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:18:59.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:59.105 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:18:59.163 INFO:teuthology.orchestra.run.smithi116.stdout:46 2023-11-08T08:18:59.163 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [46] 2023-11-08T08:18:59.164 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/) 2023-11-08T08:18:59.205 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:18:59.231 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:18:59.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:18:59.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi116.stderr:2023-11-08T08:18:59.288+0000 7fa5677ef5c0 -1 init, newargv = 0x55b289ac5d70 newargc=15 2023-11-08T08:18:59.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi116.stderr:ceph-fuse[131575]: starting ceph client 2023-11-08T08:18:59.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi116.stderr:ceph-fuse[131575]: starting fuse 2023-11-08T08:18:59.298 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:18:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:18:59.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:59.300 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:18:59.357 INFO:teuthology.orchestra.run.smithi116.stdout:46 2023-11-08T08:18:59.358 INFO:teuthology.orchestra.run.smithi116.stdout:48 2023-11-08T08:18:59.358 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46, 48] 2023-11-08T08:18:59.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:18:59.359 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-08T08:18:59.359 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-08T08:18:59.360 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:18:59.360 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-08T08:18:59.360 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:59.361 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-08T08:18:59.361 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-11-08T08:18:59.361 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-08T08:18:59.362 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-11-08T08:18:59.362 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:59.362 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-08T08:18:59.363 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-08T08:18:59.363 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-08T08:18:59.363 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-08T08:18:59.364 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:59.365 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-08T08:18:59.365 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:18:59.366 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:18:59.366 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:18:59.366 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-08T08:18:59.367 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-08T08:18:59.367 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-08T08:18:59.367 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:18:59.368 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:18:59.368 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.2")) 2023-11-08T08:18:59.368 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:18:59.479 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.2.131575.asok 2023-11-08T08:18:59.662 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.131575.asok 2023-11-08T08:18:59.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:18:59.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.131575.asok status 2023-11-08T08:18:59.817 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:18:59.818 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-08T08:18:59.818 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:18:59.818 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:18:59.819 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "2", 2023-11-08T08:18:59.819 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-08T08:18:59.819 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-11-08T08:18:59.819 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "131575", 2023-11-08T08:18:59.819 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-08T08:18:59.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:59.820 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-08T08:18:59.820 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-08T08:18:59.820 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4674, 2023-11-08T08:18:59.821 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-08T08:18:59.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-08T08:18:59.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-08T08:18:59.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4674 2023-11-08T08:18:59.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:59.822 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:18:59.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:18:59.822 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.2:0", 2023-11-08T08:18:59.822 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3147096351 2023-11-08T08:18:59.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:18:59.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:59.823 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:18:59.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:18:59.823 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.2:0", 2023-11-08T08:18:59.824 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3147096351 2023-11-08T08:18:59.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:18:59.824 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.4674 192.168.0.2:0/3147096351", 2023-11-08T08:18:59.824 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.2:0/3147096351", 2023-11-08T08:18:59.824 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-08T08:18:59.825 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 8, 2023-11-08T08:18:59.825 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 20, 2023-11-08T08:18:59.825 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:18:59.826 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-08T08:18:59.826 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-08T08:18:59.826 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:18:59.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:18:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.946+0000 7f41f552f700 1 Processor -- start 2023-11-08T08:18:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.946+0000 7f41f552f700 1 -- start start 2023-11-08T08:18:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41f552f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 0x7f41f00cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41f552f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00d1ee0 con 0x7f41f006ab10 2023-11-08T08:18:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41f552f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00cdb10 con 0x7f41f00cd5a0 2023-11-08T08:18:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41f552f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00d1890 con 0x7f41f00b22a0 2023-11-08T08:18:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41eeffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f41f006ab10 0x7f41f006aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51688/0 (socket says 172.21.15.116:51688) 2023-11-08T08:18:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41eeffd700 1 -- 172.21.15.116:0/1244149819 learned_addr learned my addr 172.21.15.116:0/1244149819 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:18:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41ef7fe700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 0x7f41f00cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119742422 0 0) 0x7f41f00d1ee0 con 0x7f41f006ab10 2023-11-08T08:18:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d80035b0 con 0x7f41f006ab10 2023-11-08T08:18:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642165297 0 0) 0x7f41d80035b0 con 0x7f41f006ab10 2023-11-08T08:18:59.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 msgr2=0x7f41f00cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:59.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 0x7f41f00cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:59.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 >> v1:172.21.15.129:6789/0 conn(0x7f41f00cd5a0 legacy=0x7f41f00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:59.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00d1b70 con 0x7f41f006ab10 2023-11-08T08:18:59.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f41e0001670 con 0x7f41f006ab10 2023-11-08T08:18:59.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 >> v1:172.21.15.116:6789/0 conn(0x7f41f006ab10 legacy=0x7f41f006aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:59.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 0x7f41d8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41d8003fb0 0x7f41d8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41d8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00d1890 con 0x7f41d8004d90 2023-11-08T08:18:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d800bba0 con 0x7f41d8003fb0 2023-11-08T08:18:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d800bd60 con 0x7f41f00b22a0 2023-11-08T08:18:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.947+0000 7f41ef7fe700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41d8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41eeffd700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 0x7f41d8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ee7fc700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41d8003fb0 0x7f41d8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 -- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 msgr2=0x7f41d8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00b22a0 0x7f41d8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 -- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41d8003fb0 msgr2=0x7f41d8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:18:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41d8003fb0 0x7f41d8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 -- 172.21.15.116:0/1244149819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80035b0 con 0x7f41d8004d90 2023-11-08T08:18:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ee7fc700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41d8003fb0 0x7f41d8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:18:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41d8005200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009f90 tx=0x7f41d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6990d19d1f21a2e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e800ec30 con 0x7f41d8004d90 2023-11-08T08:18:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e800f040 con 0x7f41d8004d90 2023-11-08T08:18:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41edffb700 1 -- 172.21.15.116:0/1244149819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8014480 con 0x7f41d8004d90 2023-11-08T08:18:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41f552f700 1 -- 172.21.15.116:0/1244149819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 msgr2=0x7f41d8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41f552f700 1 --2- 172.21.15.116:0/1244149819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41d8005200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009f90 tx=0x7f41d8005880 comp rx=0 tx=0).stop 2023-11-08T08:18:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.948+0000 7f41ef7fe700 1 -- 172.21.15.116:0/1244149819 reap_dead start 2023-11-08T08:18:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.949+0000 7f41f552f700 1 -- 172.21.15.116:0/1244149819 shutdown_connections 2023-11-08T08:18:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.949+0000 7f41f552f700 1 -- 172.21.15.116:0/1244149819 >> 172.21.15.116:0/1244149819 conn(0x7f41f006c520 msgr2=0x7f41f006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:18:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.949+0000 7f41f552f700 1 -- 172.21.15.116:0/1244149819 shutdown_connections 2023-11-08T08:18:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.949+0000 7f41f552f700 1 -- 172.21.15.116:0/1244149819 wait complete. 2023-11-08T08:18:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.950+0000 7f41f552f700 1 Processor -- start 2023-11-08T08:18:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.950+0000 7f41f552f700 1 -- start start 2023-11-08T08:18:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41f552f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41f00cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41f552f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f015aa30 0x7f41f015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41f552f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00c7e60 0x7f41f00c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41f552f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f015b3e0 con 0x7f41d8004d90 2023-11-08T08:18:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41f552f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00c8c50 con 0x7f41f015aa30 2023-11-08T08:18:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41f552f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00c8f30 con 0x7f41f00c7e60 2023-11-08T08:18:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41eeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41f00cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41eeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41f00cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58840/0 (socket says 172.21.15.116:58840) 2023-11-08T08:18:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.951+0000 7f41eeffd700 1 -- 172.21.15.116:0/667342016 learned_addr learned my addr 172.21.15.116:0/667342016 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:18:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41ee7fc700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f015aa30 0x7f41f015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41ef7fe700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00c7e60 0x7f41f00c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41eeffd700 1 -- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00c7e60 msgr2=0x7f41f00c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:59.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41eeffd700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00c7e60 0x7f41f00c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:59.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41eeffd700 1 -- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f015aa30 msgr2=0x7f41f015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:18:59.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41eeffd700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f015aa30 0x7f41f015aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:18:59.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41eeffd700 1 -- 172.21.15.116:0/667342016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00b1ec0 con 0x7f41d8004d90 2023-11-08T08:18:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41eeffd700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41f00cf6b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f41e00072c0 tx=0x7f41e0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e000f8d0 con 0x7f41d8004d90 2023-11-08T08:18:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e000fa30 con 0x7f41d8004d90 2023-11-08T08:18:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f015cad0 con 0x7f41d8004d90 2023-11-08T08:18:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e000fde0 con 0x7f41d8004d90 2023-11-08T08:18:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.952+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f00b2050 con 0x7f41d8004d90 2023-11-08T08:18:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.953+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f0158e20 con 0x7f41d8004d90 2023-11-08T08:18:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.953+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f41e001d460 con 0x7f41d8004d90 2023-11-08T08:18:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.954+0000 7f41d77fe700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41dc0682f0 0x7f41dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:18:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.954+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f41e0085ea0 con 0x7f41d8004d90 2023-11-08T08:18:59.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.954+0000 7f41ee7fc700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41dc0682f0 0x7f41dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:18:59.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.954+0000 7f41ee7fc700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41dc0682f0 0x7f41dc06a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f41f01515b0 tx=0x7f41e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:18:59.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:18:59.958+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41f0158e20 con 0x7f41d8004d90 2023-11-08T08:19:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.179+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f41f0158e20 con 0x7f41d8004d90 2023-11-08T08:19:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.180+0000 7f41d77fe700 1 -- 172.21.15.116:0/667342016 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f41f0158e20 con 0x7f41d8004d90 2023-11-08T08:19:00.181 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:19:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.181+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41dc0682f0 msgr2=0x7f41dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41dc0682f0 0x7f41dc06a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f41f01515b0 tx=0x7f41e4009040 comp rx=0 tx=0).stop 2023-11-08T08:19:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 msgr2=0x7f41f00cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41f00cf6b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f41e00072c0 tx=0x7f41e0007680 comp rx=0 tx=0).stop 2023-11-08T08:19:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 shutdown_connections 2023-11-08T08:19:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41f00c7e60 0x7f41f00c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41d8004d90 0x7f41f00cf6b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41dc0682f0 0x7f41dc06a7a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 --2- 172.21.15.116:0/667342016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f015aa30 0x7f41f015aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 >> 172.21.15.116:0/667342016 conn(0x7f41f006c520 msgr2=0x7f41f0152420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 shutdown_connections 2023-11-08T08:19:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:00.182+0000 7f41f552f700 1 -- 172.21.15.116:0/667342016 wait complete. 2023-11-08T08:19:00.194 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:19:00.195 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:19:00.195 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2023-11-08T08:19:00.195 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-08T08:19:00.195 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:00.196 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:19:00.196 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-08T08:19:00.196 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:19:00.196 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-08T08:19:00.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.197 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2023-11-08T08:19:00.219 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:19:00.219 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:19:00.220 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:19:00.220 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:00.220 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2023-11-08T08:19:00.220 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:00.220 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2023-11-08T08:19:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2023-11-08T08:19:00.222 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:00.222 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:00.222 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:19:00.222 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:00.222 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:19:00.222 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42276sec preferred_lft 42276sec 2023-11-08T08:19:00.223 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link 2023-11-08T08:19:00.223 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:00.223 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-08T08:19:00.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.224 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:00.224 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:00.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add name ceph-brx type bridge 2023-11-08T08:19:00.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip addr flush dev ceph-brx 2023-11-08T08:19:00.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set ceph-brx up 2023-11-08T08:19:00.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-08T08:19:00.225 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:00.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.386 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:19:00.414 INFO:teuthology.orchestra.run.smithi129.stdout:1 2023-11-08T08:19:00.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.416 DEBUG:teuthology.orchestra.run.smithi129:> route 2023-11-08T08:19:00.491 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2023-11-08T08:19:00.491 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:19:00.491 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:19:00.491 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:19:00.492 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:19:00.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.492 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:00.492 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:00.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:19:00.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:19:00.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:19:00.494 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:00.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.773 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2023-11-08T08:19:00.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.791 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list-id 2023-11-08T08:19:00.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.849 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:00.849 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:00.849 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-08T08:19:00.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-08T08:19:00.850 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:00.943 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-08T08:19:00.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:00.943 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:00.944 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:00.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-08T08:19:00.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-08T08:19:00.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-08T08:19:00.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-08T08:19:00.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-08T08:19:00.945 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:01.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:01.141 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:01.141 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:01.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2023-11-08T08:19:01.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set dev brx.0 master ceph-brx 2023-11-08T08:19:01.142 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:01.239 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-11-08T08:19:01.239 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:19:01.240 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:01.257 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-08T08:19:01.257 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:19:01.258 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:01.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo modprobe fuse 2023-11-08T08:19:01.381 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:19:01.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:19:01.447 INFO:teuthology.orchestra.run.smithi129.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:19:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:19:01.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:01.449 DEBUG:teuthology.orchestra.run.smithi129:> ls /sys/fs/fuse/connections 2023-11-08T08:19:01.506 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-08T08:19:01.507 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/) 2023-11-08T08:19:01.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo modprobe fuse 2023-11-08T08:19:01.575 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:19:01.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:19:01.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi129.stderr:2023-11-08T08:19:01.635+0000 7f1c6a0775c0 -1 init, newargv = 0x55adcef8a970 newargc=15 2023-11-08T08:19:01.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi129.stderr:ceph-fuse[127898]: starting ceph client 2023-11-08T08:19:01.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi129.stderr:ceph-fuse[127898]: starting fuse 2023-11-08T08:19:01.642 INFO:teuthology.orchestra.run.smithi129.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:19:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:19:01.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:01.644 DEBUG:teuthology.orchestra.run.smithi129:> ls /sys/fs/fuse/connections 2023-11-08T08:19:01.701 INFO:teuthology.orchestra.run.smithi129.stdout:46 2023-11-08T08:19:01.702 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46] 2023-11-08T08:19:01.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:01.703 DEBUG:teuthology.orchestra.run.smithi129:> import glob 2023-11-08T08:19:01.703 DEBUG:teuthology.orchestra.run.smithi129:> import re 2023-11-08T08:19:01.703 DEBUG:teuthology.orchestra.run.smithi129:> import os 2023-11-08T08:19:01.704 DEBUG:teuthology.orchestra.run.smithi129:> import subprocess 2023-11-08T08:19:01.704 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:01.704 DEBUG:teuthology.orchestra.run.smithi129:> def _find_admin_socket(client_name): 2023-11-08T08:19:01.704 DEBUG:teuthology.orchestra.run.smithi129:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-11-08T08:19:01.705 DEBUG:teuthology.orchestra.run.smithi129:> files = glob.glob(asok_path) 2023-11-08T08:19:01.705 DEBUG:teuthology.orchestra.run.smithi129:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-11-08T08:19:01.705 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:01.706 DEBUG:teuthology.orchestra.run.smithi129:> # Given a non-glob path, it better be there 2023-11-08T08:19:01.706 DEBUG:teuthology.orchestra.run.smithi129:> if "*" not in asok_path: 2023-11-08T08:19:01.706 DEBUG:teuthology.orchestra.run.smithi129:> assert(len(files) == 1) 2023-11-08T08:19:01.707 DEBUG:teuthology.orchestra.run.smithi129:> return files[0] 2023-11-08T08:19:01.707 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:01.707 DEBUG:teuthology.orchestra.run.smithi129:> for f in files: 2023-11-08T08:19:01.707 DEBUG:teuthology.orchestra.run.smithi129:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:19:01.708 DEBUG:teuthology.orchestra.run.smithi129:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:19:01.708 DEBUG:teuthology.orchestra.run.smithi129:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:19:01.708 DEBUG:teuthology.orchestra.run.smithi129:> contents = proc_f.read() 2023-11-08T08:19:01.709 DEBUG:teuthology.orchestra.run.smithi129:> if mountpoint in contents: 2023-11-08T08:19:01.709 DEBUG:teuthology.orchestra.run.smithi129:> return f 2023-11-08T08:19:01.709 DEBUG:teuthology.orchestra.run.smithi129:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:19:01.710 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:01.710 DEBUG:teuthology.orchestra.run.smithi129:> print(_find_admin_socket("client.1")) 2023-11-08T08:19:01.710 DEBUG:teuthology.orchestra.run.smithi129:> ' 2023-11-08T08:19:01.823 INFO:teuthology.orchestra.run.smithi129.stdout:/var/run/ceph/ceph-client.1.127898.asok 2023-11-08T08:19:02.007 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.127898.asok 2023-11-08T08:19:02.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.127898.asok status 2023-11-08T08:19:02.163 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-11-08T08:19:02.164 INFO:teuthology.orchestra.run.smithi129.stdout: "metadata": { 2023-11-08T08:19:02.164 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:19:02.164 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:19:02.164 INFO:teuthology.orchestra.run.smithi129.stdout: "entity_id": "1", 2023-11-08T08:19:02.165 INFO:teuthology.orchestra.run.smithi129.stdout: "hostname": "smithi129", 2023-11-08T08:19:02.165 INFO:teuthology.orchestra.run.smithi129.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-11-08T08:19:02.165 INFO:teuthology.orchestra.run.smithi129.stdout: "pid": "127898", 2023-11-08T08:19:02.165 INFO:teuthology.orchestra.run.smithi129.stdout: "root": "/" 2023-11-08T08:19:02.165 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:02.166 INFO:teuthology.orchestra.run.smithi129.stdout: "dentry_count": 0, 2023-11-08T08:19:02.166 INFO:teuthology.orchestra.run.smithi129.stdout: "dentry_pinned_count": 0, 2023-11-08T08:19:02.166 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 4439, 2023-11-08T08:19:02.166 INFO:teuthology.orchestra.run.smithi129.stdout: "inst": { 2023-11-08T08:19:02.166 INFO:teuthology.orchestra.run.smithi129.stdout: "name": { 2023-11-08T08:19:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "client", 2023-11-08T08:19:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "num": 4439 2023-11-08T08:19:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": { 2023-11-08T08:19:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2023-11-08T08:19:02.168 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:19:02.168 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 2513905695 2023-11-08T08:19:02.168 INFO:teuthology.orchestra.run.smithi129.stdout: } 2023-11-08T08:19:02.168 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:02.168 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": { 2023-11-08T08:19:02.168 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2023-11-08T08:19:02.169 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:19:02.169 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 2513905695 2023-11-08T08:19:02.169 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:02.169 INFO:teuthology.orchestra.run.smithi129.stdout: "inst_str": "client.4439 192.168.0.1:0/2513905695", 2023-11-08T08:19:02.169 INFO:teuthology.orchestra.run.smithi129.stdout: "addr_str": "192.168.0.1:0/2513905695", 2023-11-08T08:19:02.170 INFO:teuthology.orchestra.run.smithi129.stdout: "inode_count": 1, 2023-11-08T08:19:02.170 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_epoch": 8, 2023-11-08T08:19:02.170 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_epoch": 20, 2023-11-08T08:19:02.170 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:19:02.170 INFO:teuthology.orchestra.run.smithi129.stdout: "blocklisted": false, 2023-11-08T08:19:02.171 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs" 2023-11-08T08:19:02.171 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-11-08T08:19:02.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:19:02.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.305+0000 7f8139eee700 1 Processor -- start 2023-11-08T08:19:02.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.305+0000 7f8139eee700 1 -- start start 2023-11-08T08:19:02.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8139eee700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8134142db0 0x7f813413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:02.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8139eee700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813415e010 con 0x7f81340b1770 2023-11-08T08:19:02.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8139eee700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813413fc90 con 0x7f81340ce040 2023-11-08T08:19:02.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8139eee700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813413f6d0 con 0x7f8134142db0 2023-11-08T08:19:02.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8132ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8134142db0 0x7f813413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:02.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8133fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f81340b1770 0x7f81340b1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49086/0 (socket says 172.21.15.116:49086) 2023-11-08T08:19:02.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8133fff700 1 -- 172.21.15.116:0/3576176873 learned_addr learned my addr 172.21.15.116:0/3576176873 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:02.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8132ffd700 1 -- 172.21.15.116:0/3576176873 >> v1:172.21.15.129:6789/0 conn(0x7f81340ce040 legacy=0x7f813415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:02.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8132ffd700 1 -- 172.21.15.116:0/3576176873 >> v1:172.21.15.116:6789/0 conn(0x7f81340b1770 legacy=0x7f81340b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:02.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.306+0000 7f8132ffd700 1 -- 172.21.15.116:0/3576176873 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813413f9b0 con 0x7f8134142db0 2023-11-08T08:19:02.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8132ffd700 1 --2- 172.21.15.116:0/3576176873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8134142db0 0x7f813413ed70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8124009840 tx=0x7f8124009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc7082b00ba216a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:02.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f81327fc700 1 -- 172.21.15.116:0/3576176873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124004230 con 0x7f8134142db0 2023-11-08T08:19:02.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f81327fc700 1 -- 172.21.15.116:0/3576176873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8124004390 con 0x7f8134142db0 2023-11-08T08:19:02.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f81327fc700 1 -- 172.21.15.116:0/3576176873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81240036a0 con 0x7f8134142db0 2023-11-08T08:19:02.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 -- 172.21.15.116:0/3576176873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8134142db0 msgr2=0x7f813413ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:02.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 --2- 172.21.15.116:0/3576176873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8134142db0 0x7f813413ed70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8124009840 tx=0x7f8124009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:02.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 -- 172.21.15.116:0/3576176873 shutdown_connections 2023-11-08T08:19:02.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 --2- 172.21.15.116:0/3576176873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8134142db0 0x7f813413ed70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 -- 172.21.15.116:0/3576176873 >> 172.21.15.116:0/3576176873 conn(0x7f813406c010 msgr2=0x7f81340cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:02.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 -- 172.21.15.116:0/3576176873 shutdown_connections 2023-11-08T08:19:02.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.307+0000 7f8139eee700 1 -- 172.21.15.116:0/3576176873 wait complete. 2023-11-08T08:19:02.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.308+0000 7f8139eee700 1 Processor -- start 2023-11-08T08:19:02.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 -- start start 2023-11-08T08:19:02.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81340b1770 0x7f8134154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:02.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 0x7f81340c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8134142db0 0x7f8134140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340b3750 con 0x7f81340ce040 2023-11-08T08:19:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813415a0c0 con 0x7f8134142db0 2023-11-08T08:19:02.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8139eee700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813415a3a0 con 0x7f81340b1770 2023-11-08T08:19:02.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8132ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 0x7f81340c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:02.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8132ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 0x7f81340c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47682/0 (socket says 172.21.15.116:47682) 2023-11-08T08:19:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.309+0000 7f8132ffd700 1 -- 172.21.15.116:0/128774856 learned_addr learned my addr 172.21.15.116:0/128774856 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8132ffd700 1 -- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81340b1770 msgr2=0x7f8134154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8132ffd700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81340b1770 0x7f8134154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8133fff700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8134142db0 0x7f8134140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8132ffd700 1 -- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8134142db0 msgr2=0x7f8134140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8132ffd700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8134142db0 0x7f8134140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8132ffd700 1 -- 172.21.15.116:0/128774856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81240094a0 con 0x7f81340ce040 2023-11-08T08:19:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8132ffd700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 0x7f81340c5ba0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81240090b0 tx=0x7f8124004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:02.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124011630 con 0x7f81340ce040 2023-11-08T08:19:02.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812401f070 con 0x7f81340ce040 2023-11-08T08:19:02.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81240119d0 con 0x7f81340ce040 2023-11-08T08:19:02.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813415d3e0 con 0x7f81340ce040 2023-11-08T08:19:02.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.310+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813415d100 con 0x7f81340ce040 2023-11-08T08:19:02.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.311+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81340ce410 con 0x7f81340ce040 2023-11-08T08:19:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.311+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8124017020 con 0x7f81340ce040 2023-11-08T08:19:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.312+0000 7f811bfff700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81200682f0 0x7f812006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.312+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8124021070 con 0x7f81340ce040 2023-11-08T08:19:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.312+0000 7f81337fe700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81200682f0 0x7f812006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.312+0000 7f81337fe700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81200682f0 0x7f812006a7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8128001600 tx=0x7f8128007a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.316+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81240540f0 con 0x7f81340ce040 2023-11-08T08:19:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.540+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f81340743f0 con 0x7f81340ce040 2023-11-08T08:19:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.540+0000 7f811bfff700 1 -- 172.21.15.116:0/128774856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f812402b070 con 0x7f81340ce040 2023-11-08T08:19:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:19:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.542+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81200682f0 msgr2=0x7f812006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.542+0000 7f8139eee700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81200682f0 0x7f812006a7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8128001600 tx=0x7f8128007a20 comp rx=0 tx=0).stop 2023-11-08T08:19:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.542+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 msgr2=0x7f81340c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.542+0000 7f8139eee700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 0x7f81340c5ba0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81240090b0 tx=0x7f8124004750 comp rx=0 tx=0).stop 2023-11-08T08:19:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 shutdown_connections 2023-11-08T08:19:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81200682f0 0x7f812006a7a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8134142db0 0x7f8134140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81340b1770 0x7f8134154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 --2- 172.21.15.116:0/128774856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81340ce040 0x7f81340c5ba0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 >> 172.21.15.116:0/128774856 conn(0x7f813406c010 msgr2=0x7f8134157b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 shutdown_connections 2023-11-08T08:19:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:02.543+0000 7f8139eee700 1 -- 172.21.15.116:0/128774856 wait complete. 2023-11-08T08:19:02.555 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:19:02.555 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:19:02.556 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2023-11-08T08:19:02.556 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-08T08:19:02.557 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:02.557 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:19:02.557 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-08T08:19:02.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:19:02.558 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-08T08:19:02.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.560 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2023-11-08T08:19:02.579 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:19:02.579 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:19:02.580 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:19:02.580 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:02.580 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2023-11-08T08:19:02.580 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:02.580 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:02.581 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:02.581 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2023-11-08T08:19:02.581 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:02.581 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:02.581 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2023-11-08T08:19:02.582 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:02.582 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:02.582 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:19:02.582 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:02.582 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:19:02.583 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42274sec preferred_lft 42274sec 2023-11-08T08:19:02.583 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link 2023-11-08T08:19:02.583 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:02.583 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:19:02.583 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether b2:53:a4:d9:cc:2d brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-08T08:19:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::48eb:5eff:fe07:19ec/64 scope link 2023-11-08T08:19:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:02.584 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:19:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether b2:53:a4:d9:cc:2d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-08T08:19:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::b053:a4ff:fed9:cc2d/64 scope link tentative 2023-11-08T08:19:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:02.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.586 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:19:02.645 INFO:teuthology.orchestra.run.smithi129.stdout:1 2023-11-08T08:19:02.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.648 DEBUG:teuthology.orchestra.run.smithi129:> route 2023-11-08T08:19:02.708 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2023-11-08T08:19:02.708 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:19:02.708 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:19:02.709 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:19:02.709 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:19:02.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.710 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:02.710 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:02.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:19:02.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:19:02.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:19:02.711 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:02.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.812 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2023-11-08T08:19:02.830 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-08T08:19:02.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.831 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list-id 2023-11-08T08:19:02.888 INFO:teuthology.orchestra.run.smithi129.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-08T08:19:02.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.889 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:02.889 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:02.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-08T08:19:02.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-08T08:19:02.891 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:02.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:02.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-08T08:19:03.017 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:19:03.017 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:19:03.017 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:19:03.017 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.018 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2023-11-08T08:19:03.018 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.018 INFO:teuthology.orchestra.run.smithi129.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:19:03.018 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether e6:90:b9:dd:a7:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-08T08:19:03.018 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-08T08:19:03.019 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.019 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::e490:b9ff:fedd:a7dc/64 scope link tentative 2023-11-08T08:19:03.019 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:03.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-08T08:19:03.111 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:19:03.111 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:19:03.111 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:19:03.112 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.112 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2023-11-08T08:19:03.112 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.112 INFO:teuthology.orchestra.run.smithi129.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:19:03.112 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether e6:90:b9:dd:a7:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-08T08:19:03.113 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-08T08:19:03.113 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.113 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::e490:b9ff:fedd:a7dc/64 scope link tentative 2023-11-08T08:19:03.113 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:03.132 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-08T08:19:03.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:03.133 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:03.133 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:03.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-08T08:19:03.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-08T08:19:03.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-08T08:19:03.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-08T08:19:03.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-08T08:19:03.135 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:03.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:03.361 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec bash -c ' 2023-11-08T08:19:03.361 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2023-11-08T08:19:03.361 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.1 up 2023-11-08T08:19:03.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set dev brx.1 master ceph-brx 2023-11-08T08:19:03.362 DEBUG:teuthology.orchestra.run.smithi129:> ') 2023-11-08T08:19:03.458 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-11-08T08:19:03.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:19:03.459 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:03.477 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-08T08:19:03.477 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:19:03.477 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:03.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo modprobe fuse 2023-11-08T08:19:03.600 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:19:03.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:19:03.667 INFO:teuthology.orchestra.run.smithi129.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:19:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:19:03.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:03.669 DEBUG:teuthology.orchestra.run.smithi129:> ls /sys/fs/fuse/connections 2023-11-08T08:19:03.727 INFO:teuthology.orchestra.run.smithi129.stdout:46 2023-11-08T08:19:03.727 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [46] 2023-11-08T08:19:03.727 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/) 2023-11-08T08:19:03.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo modprobe fuse 2023-11-08T08:19:03.795 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:19:03.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:19:03.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi129.stderr:2023-11-08T08:19:03.854+0000 7fb1c1e775c0 -1 init, newargv = 0x556959a54910 newargc=15 2023-11-08T08:19:03.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi129.stderr:ceph-fuse[128573]: starting ceph client 2023-11-08T08:19:03.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi129.stderr:ceph-fuse[128573]: starting fuse 2023-11-08T08:19:03.865 INFO:teuthology.orchestra.run.smithi129.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:19:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:19:03.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:03.866 DEBUG:teuthology.orchestra.run.smithi129:> ls /sys/fs/fuse/connections 2023-11-08T08:19:03.924 INFO:teuthology.orchestra.run.smithi129.stdout:46 2023-11-08T08:19:03.924 INFO:teuthology.orchestra.run.smithi129.stdout:48 2023-11-08T08:19:03.924 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46, 48] 2023-11-08T08:19:03.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:03.925 DEBUG:teuthology.orchestra.run.smithi129:> import glob 2023-11-08T08:19:03.926 DEBUG:teuthology.orchestra.run.smithi129:> import re 2023-11-08T08:19:03.926 DEBUG:teuthology.orchestra.run.smithi129:> import os 2023-11-08T08:19:03.926 DEBUG:teuthology.orchestra.run.smithi129:> import subprocess 2023-11-08T08:19:03.927 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:03.927 DEBUG:teuthology.orchestra.run.smithi129:> def _find_admin_socket(client_name): 2023-11-08T08:19:03.927 DEBUG:teuthology.orchestra.run.smithi129:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-11-08T08:19:03.927 DEBUG:teuthology.orchestra.run.smithi129:> files = glob.glob(asok_path) 2023-11-08T08:19:03.928 DEBUG:teuthology.orchestra.run.smithi129:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-11-08T08:19:03.928 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:03.928 DEBUG:teuthology.orchestra.run.smithi129:> # Given a non-glob path, it better be there 2023-11-08T08:19:03.929 DEBUG:teuthology.orchestra.run.smithi129:> if "*" not in asok_path: 2023-11-08T08:19:03.929 DEBUG:teuthology.orchestra.run.smithi129:> assert(len(files) == 1) 2023-11-08T08:19:03.929 DEBUG:teuthology.orchestra.run.smithi129:> return files[0] 2023-11-08T08:19:03.930 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:03.930 DEBUG:teuthology.orchestra.run.smithi129:> for f in files: 2023-11-08T08:19:03.930 DEBUG:teuthology.orchestra.run.smithi129:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:19:03.931 DEBUG:teuthology.orchestra.run.smithi129:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:19:03.931 DEBUG:teuthology.orchestra.run.smithi129:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:19:03.931 DEBUG:teuthology.orchestra.run.smithi129:> contents = proc_f.read() 2023-11-08T08:19:03.932 DEBUG:teuthology.orchestra.run.smithi129:> if mountpoint in contents: 2023-11-08T08:19:03.932 DEBUG:teuthology.orchestra.run.smithi129:> return f 2023-11-08T08:19:03.933 DEBUG:teuthology.orchestra.run.smithi129:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:19:03.933 DEBUG:teuthology.orchestra.run.smithi129:> 2023-11-08T08:19:03.933 DEBUG:teuthology.orchestra.run.smithi129:> print(_find_admin_socket("client.3")) 2023-11-08T08:19:03.933 DEBUG:teuthology.orchestra.run.smithi129:> ' 2023-11-08T08:19:04.045 INFO:teuthology.orchestra.run.smithi129.stdout:/var/run/ceph/ceph-client.3.128573.asok 2023-11-08T08:19:04.228 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.128573.asok 2023-11-08T08:19:04.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.128573.asok status 2023-11-08T08:19:04.385 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2023-11-08T08:19:04.385 INFO:teuthology.orchestra.run.smithi129.stdout: "metadata": { 2023-11-08T08:19:04.385 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:19:04.385 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:19:04.386 INFO:teuthology.orchestra.run.smithi129.stdout: "entity_id": "3", 2023-11-08T08:19:04.386 INFO:teuthology.orchestra.run.smithi129.stdout: "hostname": "smithi129", 2023-11-08T08:19:04.386 INFO:teuthology.orchestra.run.smithi129.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-11-08T08:19:04.386 INFO:teuthology.orchestra.run.smithi129.stdout: "pid": "128573", 2023-11-08T08:19:04.387 INFO:teuthology.orchestra.run.smithi129.stdout: "root": "/" 2023-11-08T08:19:04.387 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:04.387 INFO:teuthology.orchestra.run.smithi129.stdout: "dentry_count": 0, 2023-11-08T08:19:04.387 INFO:teuthology.orchestra.run.smithi129.stdout: "dentry_pinned_count": 0, 2023-11-08T08:19:04.387 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 4442, 2023-11-08T08:19:04.388 INFO:teuthology.orchestra.run.smithi129.stdout: "inst": { 2023-11-08T08:19:04.388 INFO:teuthology.orchestra.run.smithi129.stdout: "name": { 2023-11-08T08:19:04.388 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "client", 2023-11-08T08:19:04.388 INFO:teuthology.orchestra.run.smithi129.stdout: "num": 4442 2023-11-08T08:19:04.388 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:04.389 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": { 2023-11-08T08:19:04.389 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2023-11-08T08:19:04.389 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "192.168.0.2:0", 2023-11-08T08:19:04.389 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 1345725030 2023-11-08T08:19:04.389 INFO:teuthology.orchestra.run.smithi129.stdout: } 2023-11-08T08:19:04.390 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:04.390 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": { 2023-11-08T08:19:04.390 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2023-11-08T08:19:04.390 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "192.168.0.2:0", 2023-11-08T08:19:04.391 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 1345725030 2023-11-08T08:19:04.391 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2023-11-08T08:19:04.391 INFO:teuthology.orchestra.run.smithi129.stdout: "inst_str": "client.4442 192.168.0.2:0/1345725030", 2023-11-08T08:19:04.391 INFO:teuthology.orchestra.run.smithi129.stdout: "addr_str": "192.168.0.2:0/1345725030", 2023-11-08T08:19:04.391 INFO:teuthology.orchestra.run.smithi129.stdout: "inode_count": 1, 2023-11-08T08:19:04.392 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_epoch": 8, 2023-11-08T08:19:04.392 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_epoch": 20, 2023-11-08T08:19:04.392 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:19:04.392 INFO:teuthology.orchestra.run.smithi129.stdout: "blocklisted": false, 2023-11-08T08:19:04.392 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs" 2023-11-08T08:19:04.392 INFO:teuthology.orchestra.run.smithi129.stdout:} 2023-11-08T08:19:04.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.395 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-08T08:19:04.395 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:04.418 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-08T08:19:04.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:04.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:04.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.492 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-08T08:19:04.492 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.2 2023-11-08T08:19:04.552 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-08T08:19:04.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-11-08T08:19:04.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-08T08:19:04.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.625 DEBUG:teuthology.orchestra.run.smithi129:> stat --file-system '--printf=%T 2023-11-08T08:19:04.625 DEBUG:teuthology.orchestra.run.smithi129:> ' -- /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:04.644 INFO:teuthology.orchestra.run.smithi129.stdout:fuseblk 2023-11-08T08:19:04.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:04.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:04.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.712 DEBUG:teuthology.orchestra.run.smithi129:> stat --file-system '--printf=%T 2023-11-08T08:19:04.712 DEBUG:teuthology.orchestra.run.smithi129:> ' -- /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:04.770 INFO:teuthology.orchestra.run.smithi129.stdout:fuseblk 2023-11-08T08:19:04.770 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:04.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:04.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:04.836 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-08T08:19:04.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.984+0000 7f2e5cfc4700 1 Processor -- start 2023-11-08T08:19:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.984+0000 7f2e5cfc4700 1 -- start start 2023-11-08T08:19:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5cfc4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e580b4660 0x7f2e58071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5cfc4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580cfb00 con 0x7f2e580b4a30 2023-11-08T08:19:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5cfc4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5815aa20 con 0x7f2e580cff90 2023-11-08T08:19:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5cfc4700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e580cf4d0 con 0x7f2e580b4660 2023-11-08T08:19:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5659c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e580b4660 0x7f2e58071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e56d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2e580b4a30 0x7f2e5815f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49100/0 (socket says 172.21.15.116:49100) 2023-11-08T08:19:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e56d9d700 1 -- 172.21.15.116:0/2666191679 learned_addr learned my addr 172.21.15.116:0/2666191679 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5659c700 1 -- 172.21.15.116:0/2666191679 >> v1:172.21.15.129:6789/0 conn(0x7f2e580cff90 legacy=0x7f2e5806b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5659c700 1 -- 172.21.15.116:0/2666191679 >> v1:172.21.15.116:6789/0 conn(0x7f2e580b4a30 legacy=0x7f2e5815f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5659c700 1 -- 172.21.15.116:0/2666191679 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e58072550 con 0x7f2e580b4660 2023-11-08T08:19:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.985+0000 7f2e5559a700 1 -- 172.21.15.116:0/2666191679 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078977953 0 0) 0x7f2e580cfb00 con 0x7f2e580b4a30 2023-11-08T08:19:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5659c700 1 --2- 172.21.15.116:0/2666191679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e58071f70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009840 tx=0x7f2e4c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f14501da1172e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5559a700 1 -- 172.21.15.116:0/2666191679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c011830 con 0x7f2e580b4660 2023-11-08T08:19:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5559a700 1 -- 172.21.15.116:0/2666191679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c011990 con 0x7f2e580b4660 2023-11-08T08:19:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5559a700 1 -- 172.21.15.116:0/2666191679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c011d20 con 0x7f2e580b4660 2023-11-08T08:19:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/2666191679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 msgr2=0x7f2e58071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/2666191679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e58071f70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009840 tx=0x7f2e4c009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/2666191679 shutdown_connections 2023-11-08T08:19:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/2666191679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e58071f70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/2666191679 >> 172.21.15.116:0/2666191679 conn(0x7f2e580b2fc0 msgr2=0x7f2e58071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/2666191679 shutdown_connections 2023-11-08T08:19:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.986+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/2666191679 wait complete. 2023-11-08T08:19:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.988+0000 7f2e5cfc4700 1 Processor -- start 2023-11-08T08:19:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.988+0000 7f2e5cfc4700 1 -- start start 2023-11-08T08:19:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.988+0000 7f2e5cfc4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e580acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.988+0000 7f2e5cfc4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e580b4a30 0x7f2e58156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.988+0000 7f2e5cfc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e580cff90 0x7f2e580c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5cfc4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580c6bc0 con 0x7f2e580b4660 2023-11-08T08:19:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5cfc4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580c70d0 con 0x7f2e580cff90 2023-11-08T08:19:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5cfc4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580c73b0 con 0x7f2e580b4a30 2023-11-08T08:19:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e580acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e580acbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47694/0 (socket says 172.21.15.116:47694) 2023-11-08T08:19:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 -- 172.21.15.116:0/714796520 learned_addr learned my addr 172.21.15.116:0/714796520 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 -- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e580b4a30 msgr2=0x7f2e58156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e580b4a30 0x7f2e58156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 -- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e580cff90 msgr2=0x7f2e580c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e56d9d700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e580cff90 0x7f2e580c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e580cff90 0x7f2e580c89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 -- 172.21.15.116:0/714796520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0094a0 con 0x7f2e580b4660 2023-11-08T08:19:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5659c700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e580acbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009b90 tx=0x7f2e4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0036a0 con 0x7f2e580b4660 2023-11-08T08:19:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c01e070 con 0x7f2e580b4660 2023-11-08T08:19:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5815f880 con 0x7f2e580b4660 2023-11-08T08:19:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c003a00 con 0x7f2e580b4660 2023-11-08T08:19:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.989+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e580cf790 con 0x7f2e580b4660 2023-11-08T08:19:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.990+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5815fa10 con 0x7f2e580b4660 2023-11-08T08:19:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.991+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e4c00f050 con 0x7f2e580b4660 2023-11-08T08:19:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.991+0000 7f2e46ffd700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e380682f0 0x7f2e3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.991+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2e4c086df0 con 0x7f2e580b4660 2023-11-08T08:19:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.991+0000 7f2e55d9b700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e380682f0 0x7f2e3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.991+0000 7f2e55d9b700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e380682f0 0x7f2e3806a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e48000f60 tx=0x7f2e48007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:04.995+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e4c053650 con 0x7f2e580b4660 2023-11-08T08:19:05.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.176+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e580743f0 con 0x7f2e580b4660 2023-11-08T08:19:05.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.177+0000 7f2e46ffd700 1 -- 172.21.15.116:0/714796520 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f2e4c015090 con 0x7f2e580b4660 2023-11-08T08:19:05.178 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:05.179 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:05.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e380682f0 msgr2=0x7f2e3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e380682f0 0x7f2e3806a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e48000f60 tx=0x7f2e48007a40 comp rx=0 tx=0).stop 2023-11-08T08:19:05.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 msgr2=0x7f2e580acbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e580acbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009b90 tx=0x7f2e4c004750 comp rx=0 tx=0).stop 2023-11-08T08:19:05.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 shutdown_connections 2023-11-08T08:19:05.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2e380682f0 0x7f2e3806a7a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e580cff90 0x7f2e580c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e580b4660 0x7f2e580acbc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 --2- 172.21.15.116:0/714796520 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e580b4a30 0x7f2e58156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 >> 172.21.15.116:0/714796520 conn(0x7f2e580b2fc0 msgr2=0x7f2e5806c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:05.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 shutdown_connections 2023-11-08T08:19:05.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.179+0000 7f2e5cfc4700 1 -- 172.21.15.116:0/714796520 wait complete. 2023-11-08T08:19:05.192 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:05.192 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.314+0000 7fce48c9f700 1 Processor -- start 2023-11-08T08:19:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.314+0000 7fce48c9f700 1 -- start start 2023-11-08T08:19:05.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce48c9f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fce440718e0 0x7fce4406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce48c9f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce4406cdc0 0x7fce4406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce48c9f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce44160a50 con 0x7fce4415c810 2023-11-08T08:19:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce48c9f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4415cbe0 con 0x7fce4406cdc0 2023-11-08T08:19:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce48c9f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce44160d30 con 0x7fce440718e0 2023-11-08T08:19:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce4406cdc0 0x7fce4406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce4406cdc0 0x7fce4406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47708/0 (socket says 172.21.15.116:47708) 2023-11-08T08:19:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 -- 172.21.15.116:0/3669842921 learned_addr learned my addr 172.21.15.116:0/3669842921 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce41d9b700 1 --2- 172.21.15.116:0/3669842921 >> v2:172.21.15.129:3300/0 conn(0x7fce440718e0 0x7fce4406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 -- 172.21.15.116:0/3669842921 >> v2:172.21.15.129:3300/0 conn(0x7fce440718e0 msgr2=0x7fce4406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 --2- 172.21.15.116:0/3669842921 >> v2:172.21.15.129:3300/0 conn(0x7fce440718e0 0x7fce4406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 -- 172.21.15.116:0/3669842921 >> v1:172.21.15.129:6789/0 conn(0x7fce4415c810 legacy=0x7fce44160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 -- 172.21.15.116:0/3669842921 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce44161010 con 0x7fce4406cdc0 2023-11-08T08:19:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce41d9b700 1 --2- 172.21.15.116:0/3669842921 >> v2:172.21.15.129:3300/0 conn(0x7fce440718e0 0x7fce4406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce42d9d700 1 --2- 172.21.15.116:0/3669842921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4406cdc0 0x7fce4406a270 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009e50 tx=0x7fce3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5bb3987d6a5bfe server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce4159a700 1 -- 172.21.15.116:0/3669842921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c00e070 con 0x7fce4406cdc0 2023-11-08T08:19:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.315+0000 7fce4159a700 1 -- 172.21.15.116:0/3669842921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3c003a20 con 0x7fce4406cdc0 2023-11-08T08:19:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce4159a700 1 -- 172.21.15.116:0/3669842921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c003dd0 con 0x7fce4406cdc0 2023-11-08T08:19:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 -- 172.21.15.116:0/3669842921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4406cdc0 msgr2=0x7fce4406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 --2- 172.21.15.116:0/3669842921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4406cdc0 0x7fce4406a270 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009e50 tx=0x7fce3c00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 -- 172.21.15.116:0/3669842921 shutdown_connections 2023-11-08T08:19:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 --2- 172.21.15.116:0/3669842921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4406cdc0 0x7fce4406a270 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 --2- 172.21.15.116:0/3669842921 >> v2:172.21.15.129:3300/0 conn(0x7fce440718e0 0x7fce4406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 -- 172.21.15.116:0/3669842921 >> 172.21.15.116:0/3669842921 conn(0x7fce4406c010 msgr2=0x7fce440cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 -- 172.21.15.116:0/3669842921 shutdown_connections 2023-11-08T08:19:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.316+0000 7fce48c9f700 1 -- 172.21.15.116:0/3669842921 wait complete. 2023-11-08T08:19:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.317+0000 7fce48c9f700 1 Processor -- start 2023-11-08T08:19:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.317+0000 7fce48c9f700 1 -- start start 2023-11-08T08:19:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce48c9f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fce4406cdc0 0x7fce44157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce48c9f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 0x7fce4415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce48c9f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce4415c810 0x7fce4414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce48c9f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce44158210 con 0x7fce440718e0 2023-11-08T08:19:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce48c9f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce441577d0 con 0x7fce4415c810 2023-11-08T08:19:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce48c9f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce440b0770 con 0x7fce4406cdc0 2023-11-08T08:19:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 0x7fce4415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 0x7fce4415b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47710/0 (socket says 172.21.15.116:47710) 2023-11-08T08:19:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.318+0000 7fce41d9b700 1 -- 172.21.15.116:0/319743917 learned_addr learned my addr 172.21.15.116:0/319743917 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce42d9d700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce4415c810 0x7fce4414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce4259c700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fce4406cdc0 0x7fce44157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce41d9b700 1 -- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fce4406cdc0 msgr2=0x7fce44157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce41d9b700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fce4406cdc0 0x7fce44157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce41d9b700 1 -- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce4415c810 msgr2=0x7fce4414ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce41d9b700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce4415c810 0x7fce4414ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce41d9b700 1 -- 172.21.15.116:0/319743917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c009b30 con 0x7fce440718e0 2023-11-08T08:19:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce42d9d700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce4415c810 0x7fce4414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce4259c700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fce4406cdc0 0x7fce44157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce41d9b700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 0x7fce4415b0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fce3800c750 tx=0x7fce3800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38011800 con 0x7fce440718e0 2023-11-08T08:19:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce38011960 con 0x7fce440718e0 2023-11-08T08:19:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce440ce740 con 0x7fce440718e0 2023-11-08T08:19:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38011cd0 con 0x7fce440718e0 2023-11-08T08:19:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.319+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce440ceb20 con 0x7fce440718e0 2023-11-08T08:19:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.320+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce440ce8d0 con 0x7fce440718e0 2023-11-08T08:19:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.320+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce38020070 con 0x7fce440718e0 2023-11-08T08:19:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.321+0000 7fce32ffd700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fce240682f0 0x7fce2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.321+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fce38085b10 con 0x7fce440718e0 2023-11-08T08:19:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.321+0000 7fce4259c700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fce240682f0 0x7fce2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.321+0000 7fce4259c700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fce240682f0 0x7fce2406a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fce34006fd0 tx=0x7fce34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.325+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce38052370 con 0x7fce440718e0 2023-11-08T08:19:05.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.507+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fce440743f0 con 0x7fce440718e0 2023-11-08T08:19:05.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.507+0000 7fce32ffd700 1 -- 172.21.15.116:0/319743917 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fce38055990 con 0x7fce440718e0 2023-11-08T08:19:05.509 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:05.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fce240682f0 msgr2=0x7fce2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fce240682f0 0x7fce2406a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fce34006fd0 tx=0x7fce34008040 comp rx=0 tx=0).stop 2023-11-08T08:19:05.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 msgr2=0x7fce4415b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 0x7fce4415b0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fce3800c750 tx=0x7fce3800cb10 comp rx=0 tx=0).stop 2023-11-08T08:19:05.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 shutdown_connections 2023-11-08T08:19:05.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fce240682f0 0x7fce2406a7a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce4415c810 0x7fce4414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fce4406cdc0 0x7fce44157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 --2- 172.21.15.116:0/319743917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce440718e0 0x7fce4415b0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 >> 172.21.15.116:0/319743917 conn(0x7fce4406c010 msgr2=0x7fce440b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 shutdown_connections 2023-11-08T08:19:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.509+0000 7fce48c9f700 1 -- 172.21.15.116:0/319743917 wait complete. 2023-11-08T08:19:05.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:05.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.644+0000 7f9686fa6700 1 Processor -- start 2023-11-08T08:19:05.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.644+0000 7f9686fa6700 1 -- start start 2023-11-08T08:19:05.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9686fa6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f96800cec00 0x7f9680065fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9686fa6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9680160300 0x7f968015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9686fa6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968015ceb0 con 0x7f968015d880 2023-11-08T08:19:05.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9686fa6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96801608c0 con 0x7f9680160300 2023-11-08T08:19:05.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9686fa6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9680160ba0 con 0x7f96800cec00 2023-11-08T08:19:05.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9680160300 0x7f968015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9680160300 0x7f968015d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47712/0 (socket says 172.21.15.116:47712) 2023-11-08T08:19:05.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 -- 172.21.15.116:0/490583265 learned_addr learned my addr 172.21.15.116:0/490583265 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:05.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 -- 172.21.15.116:0/490583265 >> v2:172.21.15.129:3300/0 conn(0x7f96800cec00 msgr2=0x7f9680065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9684d42700 1 --2- 172.21.15.116:0/490583265 >> v2:172.21.15.129:3300/0 conn(0x7f96800cec00 0x7f9680065fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 --2- 172.21.15.116:0/490583265 >> v2:172.21.15.129:3300/0 conn(0x7f96800cec00 0x7f9680065fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 -- 172.21.15.116:0/490583265 >> v1:172.21.15.129:6789/0 conn(0x7f968015d880 legacy=0x7f96800cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f967ffff700 1 -- 172.21.15.116:0/490583265 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9680160e80 con 0x7f9680160300 2023-11-08T08:19:05.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.645+0000 7f9684d42700 1 --2- 172.21.15.116:0/490583265 >> v2:172.21.15.129:3300/0 conn(0x7f96800cec00 0x7f9680065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f967ffff700 1 --2- 172.21.15.116:0/490583265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680160300 0x7f968015d340 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f96800cceb0 tx=0x7f966c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33961de7bab31547 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f967f7fe700 1 -- 172.21.15.116:0/490583265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c004230 con 0x7f9680160300 2023-11-08T08:19:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f967f7fe700 1 -- 172.21.15.116:0/490583265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c004390 con 0x7f9680160300 2023-11-08T08:19:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f967f7fe700 1 -- 172.21.15.116:0/490583265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c0036a0 con 0x7f9680160300 2023-11-08T08:19:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 -- 172.21.15.116:0/490583265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680160300 msgr2=0x7f968015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 --2- 172.21.15.116:0/490583265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680160300 0x7f968015d340 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f96800cceb0 tx=0x7f966c009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 -- 172.21.15.116:0/490583265 shutdown_connections 2023-11-08T08:19:05.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 --2- 172.21.15.116:0/490583265 >> v2:172.21.15.129:3300/0 conn(0x7f96800cec00 0x7f9680065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 --2- 172.21.15.116:0/490583265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680160300 0x7f968015d340 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 -- 172.21.15.116:0/490583265 >> 172.21.15.116:0/490583265 conn(0x7f96800cf400 msgr2=0x7f96800cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:05.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 -- 172.21.15.116:0/490583265 shutdown_connections 2023-11-08T08:19:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.646+0000 7f9686fa6700 1 -- 172.21.15.116:0/490583265 wait complete. 2023-11-08T08:19:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.648+0000 7f9686fa6700 1 Processor -- start 2023-11-08T08:19:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.648+0000 7f9686fa6700 1 -- start start 2023-11-08T08:19:05.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9686fa6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 0x7f96800c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9686fa6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f968015d880 0x7f96800c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9686fa6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9680160300 0x7f9680071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9686fa6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800c53f0 con 0x7f96800cec00 2023-11-08T08:19:05.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9686fa6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800c47e0 con 0x7f968015d880 2023-11-08T08:19:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9686fa6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96800c4ac0 con 0x7f9680160300 2023-11-08T08:19:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 0x7f96800c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 0x7f96800c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47718/0 (socket says 172.21.15.116:47718) 2023-11-08T08:19:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 -- 172.21.15.116:0/458205297 learned_addr learned my addr 172.21.15.116:0/458205297 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:05.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 -- 172.21.15.116:0/458205297 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9680160300 msgr2=0x7f9680071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9680160300 0x7f9680071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 -- 172.21.15.116:0/458205297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f968015d880 msgr2=0x7f96800c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f968015d880 0x7f96800c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 -- 172.21.15.116:0/458205297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c009550 con 0x7f96800cec00 2023-11-08T08:19:05.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.649+0000 7f9684d42700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 0x7f96800c8d20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f967400c750 tx=0x7f967400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.650+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674011800 con 0x7f96800cec00 2023-11-08T08:19:05.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.650+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9674011960 con 0x7f96800cec00 2023-11-08T08:19:05.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.650+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674011cd0 con 0x7f96800cec00 2023-11-08T08:19:05.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.650+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9680066510 con 0x7f96800cec00 2023-11-08T08:19:05.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.650+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9680161160 con 0x7f96800cec00 2023-11-08T08:19:05.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.650+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96800cefd0 con 0x7f96800cec00 2023-11-08T08:19:05.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.651+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9674020070 con 0x7f96800cec00 2023-11-08T08:19:05.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.651+0000 7f967d7fa700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96640682f0 0x7f966406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.651+0000 7f967ffff700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96640682f0 0x7f966406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.652+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9674085b10 con 0x7f96800cec00 2023-11-08T08:19:05.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.652+0000 7f967ffff700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96640682f0 0x7f966406a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f96801f5a10 tx=0x7f966c010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.656+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9674052370 con 0x7f96800cec00 2023-11-08T08:19:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.834+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96800c56d0 con 0x7f96800cec00 2023-11-08T08:19:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.835+0000 7f967d7fa700 1 -- 172.21.15.116:0/458205297 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9674055990 con 0x7f96800cec00 2023-11-08T08:19:05.836 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96640682f0 msgr2=0x7f966406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96640682f0 0x7f966406a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f96801f5a10 tx=0x7f966c010000 comp rx=0 tx=0).stop 2023-11-08T08:19:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 msgr2=0x7f96800c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 0x7f96800c8d20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f967400c750 tx=0x7f967400cb10 comp rx=0 tx=0).stop 2023-11-08T08:19:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 shutdown_connections 2023-11-08T08:19:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9680160300 0x7f9680071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96800cec00 0x7f96800c8d20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96640682f0 0x7f966406a7a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 --2- 172.21.15.116:0/458205297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f968015d880 0x7f96800c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 >> 172.21.15.116:0/458205297 conn(0x7f96800cf400 msgr2=0x7f96800af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 shutdown_connections 2023-11-08T08:19:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.837+0000 7f9686fa6700 1 -- 172.21.15.116:0/458205297 wait complete. 2023-11-08T08:19:05.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.972+0000 7faa9417d700 1 Processor -- start 2023-11-08T08:19:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.972+0000 7faa9417d700 1 -- start start 2023-11-08T08:19:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa9417d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7faa8c0b0540 0x7faa8c0c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa9417d700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0ca7b0 con 0x7faa8c15e910 2023-11-08T08:19:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa9417d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c15a8c0 con 0x7faa8c0ca310 2023-11-08T08:19:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa9417d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c0caa10 con 0x7faa8c0b0540 2023-11-08T08:19:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa9271a700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faa8c15e910 0x7faa8c070940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49108/0 (socket says 172.21.15.116:49108) 2023-11-08T08:19:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa9271a700 1 -- 172.21.15.116:0/4211224314 learned_addr learned my addr 172.21.15.116:0/4211224314 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054537657 0 0) 0x7faa8c0ca7b0 con 0x7faa8c15e910 2023-11-08T08:19:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa91f19700 1 --2- 172.21.15.116:0/4211224314 >> v2:172.21.15.129:3300/0 conn(0x7faa8c0b0540 0x7faa8c0c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa780035b0 con 0x7faa8c15e910 2023-11-08T08:19:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 74466981 0 0) 0x7faa780035b0 con 0x7faa8c15e910 2023-11-08T08:19:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 >> v2:172.21.15.129:3300/0 conn(0x7faa8c0b0540 msgr2=0x7faa8c0c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 --2- 172.21.15.116:0/4211224314 >> v2:172.21.15.129:3300/0 conn(0x7faa8c0b0540 0x7faa8c0c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 >> v1:172.21.15.129:6789/0 conn(0x7faa8c0ca310 legacy=0x7faa8c0b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0cacf0 con 0x7faa8c15e910 2023-11-08T08:19:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7faa840019f0 con 0x7faa8c15e910 2023-11-08T08:19:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 >> v1:172.21.15.116:6789/0 conn(0x7faa8c15e910 legacy=0x7faa8c070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78003a00 0x7faa78003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa78004850 0x7faa78004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa78005200 0x7faa7800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0caa10 con 0x7faa78005200 2023-11-08T08:19:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7800c080 con 0x7faa78003a00 2023-11-08T08:19:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.973+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7800c240 con 0x7faa78004850 2023-11-08T08:19:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa78005200 0x7faa7800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa91f19700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78003a00 0x7faa78003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa91718700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa78004850 0x7faa78004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 -- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa78004850 msgr2=0x7faa78004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa78004850 0x7faa78004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 -- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78003a00 msgr2=0x7faa78003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78003a00 0x7faa78003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 -- 172.21.15.116:0/4211224314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780035b0 con 0x7faa78005200 2023-11-08T08:19:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 -- 172.21.15.116:0/4211224314 reap_dead start 2023-11-08T08:19:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa9271a700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa78005200 0x7faa7800ba10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faa840013d0 tx=0x7faa78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c31fd485b1107af server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84012600 con 0x7faa78005200 2023-11-08T08:19:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa840127a0 con 0x7faa78005200 2023-11-08T08:19:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.974+0000 7faa90f17700 1 -- 172.21.15.116:0/4211224314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84012b50 con 0x7faa78005200 2023-11-08T08:19:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 -- 172.21.15.116:0/4211224314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa78005200 msgr2=0x7faa7800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa78005200 0x7faa7800ba10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faa840013d0 tx=0x7faa78005880 comp rx=0 tx=0).stop 2023-11-08T08:19:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 -- 172.21.15.116:0/4211224314 shutdown_connections 2023-11-08T08:19:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 --2- 172.21.15.116:0/4211224314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa78005200 0x7faa7800ba10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 -- 172.21.15.116:0/4211224314 >> 172.21.15.116:0/4211224314 conn(0x7faa8c0cb540 msgr2=0x7faa8c0cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 -- 172.21.15.116:0/4211224314 shutdown_connections 2023-11-08T08:19:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.975+0000 7faa9417d700 1 -- 172.21.15.116:0/4211224314 wait complete. 2023-11-08T08:19:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.976+0000 7faa9417d700 1 Processor -- start 2023-11-08T08:19:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.976+0000 7faa9417d700 1 -- start start 2023-11-08T08:19:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78005200 0x7faa8c1f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 0x7faa8c0af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa8c0c7500 0x7faa8c0af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1f3a70 con 0x7faa8c0c70b0 2023-11-08T08:19:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0c7aa0 con 0x7faa78005200 2023-11-08T08:19:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0c7d80 con 0x7faa8c0c7500 2023-11-08T08:19:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 0x7faa8c0af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 0x7faa8c0af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47748/0 (socket says 172.21.15.116:47748) 2023-11-08T08:19:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 -- 172.21.15.116:0/700703561 learned_addr learned my addr 172.21.15.116:0/700703561 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91f19700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78005200 0x7faa8c1f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9271a700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa8c0c7500 0x7faa8c0af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 -- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa8c0c7500 msgr2=0x7faa8c0af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa8c0c7500 0x7faa8c0af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 -- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78005200 msgr2=0x7faa8c1f3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:05.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78005200 0x7faa8c1f3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:05.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 -- 172.21.15.116:0/700703561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0c93f0 con 0x7faa8c0c70b0 2023-11-08T08:19:05.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa91718700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 0x7faa8c0af220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00f820 tx=0x7faa7c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c0052e0 con 0x7faa8c0c70b0 2023-11-08T08:19:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c005440 con 0x7faa8c0c70b0 2023-11-08T08:19:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c0038c0 con 0x7faa8c0c70b0 2023-11-08T08:19:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c0c9110 con 0x7faa8c0c70b0 2023-11-08T08:19:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.977+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c0c96d0 con 0x7faa8c0c70b0 2023-11-08T08:19:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.978+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c0743f0 con 0x7faa8c0c70b0 2023-11-08T08:19:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.979+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa7c003a20 con 0x7faa8c0c70b0 2023-11-08T08:19:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.979+0000 7faa827fc700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faa70068320 0x7faa7006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.979+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7faa7c0859e0 con 0x7faa8c0c70b0 2023-11-08T08:19:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.979+0000 7faa91f19700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faa70068320 0x7faa7006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.979+0000 7faa91f19700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faa70068320 0x7faa7006a7d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faa88009be0 tx=0x7faa88009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:05.983+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa7c052240 con 0x7faa8c0c70b0 2023-11-08T08:19:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.160+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa8c0cc320 con 0x7faa8c0c70b0 2023-11-08T08:19:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.161+0000 7faa827fc700 1 -- 172.21.15.116:0/700703561 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa7c059e60 con 0x7faa8c0c70b0 2023-11-08T08:19:06.162 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.162+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faa70068320 msgr2=0x7faa7006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.162+0000 7faa9417d700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faa70068320 0x7faa7006a7d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faa88009be0 tx=0x7faa88009210 comp rx=0 tx=0).stop 2023-11-08T08:19:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.162+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 msgr2=0x7faa8c0af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.162+0000 7faa9417d700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 0x7faa8c0af220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00f820 tx=0x7faa7c00fb30 comp rx=0 tx=0).stop 2023-11-08T08:19:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 shutdown_connections 2023-11-08T08:19:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7faa8c0c7500 0x7faa8c0af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faa70068320 0x7faa7006a7d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa78005200 0x7faa8c1f3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 --2- 172.21.15.116:0/700703561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0c70b0 0x7faa8c0af220 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 >> 172.21.15.116:0/700703561 conn(0x7faa8c0cb540 msgr2=0x7faa8c0715f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 shutdown_connections 2023-11-08T08:19:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.163+0000 7faa9417d700 1 -- 172.21.15.116:0/700703561 wait complete. 2023-11-08T08:19:06.175 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.297+0000 7fd599492700 1 Processor -- start 2023-11-08T08:19:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.297+0000 7fd599492700 1 -- start start 2023-11-08T08:19:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd599492700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd59415c8c0 0x7fd594160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd599492700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940d2d30 con 0x7fd5940b5dc0 2023-11-08T08:19:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd599492700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940d0b40 con 0x7fd59406b220 2023-11-08T08:19:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd599492700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5941611f0 con 0x7fd59415c8c0 2023-11-08T08:19:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd592ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd59415c8c0 0x7fd594160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd5927fc700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd5940b5dc0 0x7fd59406ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49122/0 (socket says 172.21.15.116:49122) 2023-11-08T08:19:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd5927fc700 1 -- 172.21.15.116:0/1673286802 learned_addr learned my addr 172.21.15.116:0/1673286802 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd591ffb700 1 -- 172.21.15.116:0/1673286802 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608606773 0 0) 0x7fd584001110 con 0x7fd5940b5dc0 2023-11-08T08:19:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd591ffb700 1 -- 172.21.15.116:0/1673286802 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd57c0035b0 con 0x7fd5940b5dc0 2023-11-08T08:19:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd592ffd700 1 -- 172.21.15.116:0/1673286802 >> v1:172.21.15.129:6789/0 conn(0x7fd59406b220 legacy=0x7fd5940d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd592ffd700 1 -- 172.21.15.116:0/1673286802 >> v1:172.21.15.116:6789/0 conn(0x7fd5940b5dc0 legacy=0x7fd59406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.298+0000 7fd592ffd700 1 -- 172.21.15.116:0/1673286802 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940d2a50 con 0x7fd59415c8c0 2023-11-08T08:19:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd592ffd700 1 --2- 172.21.15.116:0/1673286802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59415c8c0 0x7fd594160cb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd588009840 tx=0x7fd588009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1051d4149b3cb2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd591ffb700 1 -- 172.21.15.116:0/1673286802 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588011830 con 0x7fd59415c8c0 2023-11-08T08:19:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd591ffb700 1 -- 172.21.15.116:0/1673286802 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd588011990 con 0x7fd59415c8c0 2023-11-08T08:19:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd591ffb700 1 -- 172.21.15.116:0/1673286802 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588011d00 con 0x7fd59415c8c0 2023-11-08T08:19:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 -- 172.21.15.116:0/1673286802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59415c8c0 msgr2=0x7fd594160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 --2- 172.21.15.116:0/1673286802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59415c8c0 0x7fd594160cb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd588009840 tx=0x7fd588009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 -- 172.21.15.116:0/1673286802 shutdown_connections 2023-11-08T08:19:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 --2- 172.21.15.116:0/1673286802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59415c8c0 0x7fd594160cb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 -- 172.21.15.116:0/1673286802 >> 172.21.15.116:0/1673286802 conn(0x7fd59406c520 msgr2=0x7fd59406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 -- 172.21.15.116:0/1673286802 shutdown_connections 2023-11-08T08:19:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.299+0000 7fd599492700 1 -- 172.21.15.116:0/1673286802 wait complete. 2023-11-08T08:19:06.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 Processor -- start 2023-11-08T08:19:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 -- start start 2023-11-08T08:19:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 0x7fd5941538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5940b5dc0 0x7fd5941e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59415c8c0 0x7fd594155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594153e30 con 0x7fd59406b220 2023-11-08T08:19:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941e6140 con 0x7fd59415c8c0 2023-11-08T08:19:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd599492700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941e6420 con 0x7fd5940b5dc0 2023-11-08T08:19:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd592ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 0x7fd5941538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd592ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 0x7fd5941538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47760/0 (socket says 172.21.15.116:47760) 2023-11-08T08:19:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.301+0000 7fd592ffd700 1 -- 172.21.15.116:0/1323440138 learned_addr learned my addr 172.21.15.116:0/1323440138 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd592ffd700 1 -- 172.21.15.116:0/1323440138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5940b5dc0 msgr2=0x7fd5941e7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd592ffd700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5940b5dc0 0x7fd5941e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd592ffd700 1 -- 172.21.15.116:0/1323440138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59415c8c0 msgr2=0x7fd594155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd592ffd700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59415c8c0 0x7fd594155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd592ffd700 1 -- 172.21.15.116:0/1323440138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880094a0 con 0x7fd59406b220 2023-11-08T08:19:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd592ffd700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 0x7fd5941538f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd588009fd0 tx=0x7fd588004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5880036a0 con 0x7fd59406b220 2023-11-08T08:19:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58801e070 con 0x7fd59406b220 2023-11-08T08:19:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5940cda40 con 0x7fd59406b220 2023-11-08T08:19:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588003a40 con 0x7fd59406b220 2023-11-08T08:19:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.302+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5940cd760 con 0x7fd59406b220 2023-11-08T08:19:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.303+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5940d2700 con 0x7fd59406b220 2023-11-08T08:19:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.303+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd58800f050 con 0x7fd59406b220 2023-11-08T08:19:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.304+0000 7fd57b7fe700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5800682f0 0x7fd58006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.304+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd588086ce0 con 0x7fd59406b220 2023-11-08T08:19:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.304+0000 7fd5927fc700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5800682f0 0x7fd58006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.304+0000 7fd5927fc700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5800682f0 0x7fd58006a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd584005910 tx=0x7fd584001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.308+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd588053540 con 0x7fd59406b220 2023-11-08T08:19:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.489+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd594067d70 con 0x7fd59406b220 2023-11-08T08:19:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.489+0000 7fd57b7fe700 1 -- 172.21.15.116:0/1323440138 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd5880150a0 con 0x7fd59406b220 2023-11-08T08:19:06.491 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5800682f0 msgr2=0x7fd58006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5800682f0 0x7fd58006a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd584005910 tx=0x7fd584001150 comp rx=0 tx=0).stop 2023-11-08T08:19:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 msgr2=0x7fd5941538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 0x7fd5941538f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd588009fd0 tx=0x7fd588004750 comp rx=0 tx=0).stop 2023-11-08T08:19:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 shutdown_connections 2023-11-08T08:19:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5800682f0 0x7fd58006a7a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59415c8c0 0x7fd594155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59406b220 0x7fd5941538f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 --2- 172.21.15.116:0/1323440138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5940b5dc0 0x7fd5941e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 >> 172.21.15.116:0/1323440138 conn(0x7fd59406c520 msgr2=0x7fd5940d1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 shutdown_connections 2023-11-08T08:19:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.491+0000 7fd599492700 1 -- 172.21.15.116:0/1323440138 wait complete. 2023-11-08T08:19:06.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.623+0000 7f9f357f6700 1 Processor -- start 2023-11-08T08:19:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.624+0000 7f9f357f6700 1 -- start start 2023-11-08T08:19:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.624+0000 7f9f357f6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9f3006ab10 0x7f9f3006aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.624+0000 7f9f357f6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300d1ee0 con 0x7f9f300d0450 2023-11-08T08:19:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.624+0000 7f9f357f6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f30161120 con 0x7f9f300cd5a0 2023-11-08T08:19:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.624+0000 7f9f357f6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3006b420 con 0x7f9f3006ab10 2023-11-08T08:19:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.624+0000 7f9f2f7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9f300d0450 0x7f9f30160a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49136/0 (socket says 172.21.15.116:49136) 2023-11-08T08:19:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2f7fe700 1 -- 172.21.15.116:0/3382943992 learned_addr learned my addr 172.21.15.116:0/3382943992 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3382943992 >> v2:172.21.15.129:3300/0 conn(0x7f9f3006ab10 0x7f9f3006aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293341343 0 0) 0x7f9f300d1ee0 con 0x7f9f300d0450 2023-11-08T08:19:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f180035b0 con 0x7f9f300d0450 2023-11-08T08:19:06.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052533034 0 0) 0x7f9f180035b0 con 0x7f9f300d0450 2023-11-08T08:19:06.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 >> v2:172.21.15.129:3300/0 conn(0x7f9f3006ab10 msgr2=0x7f9f3006aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 --2- 172.21.15.116:0/3382943992 >> v2:172.21.15.129:3300/0 conn(0x7f9f3006ab10 0x7f9f3006aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 >> v1:172.21.15.129:6789/0 conn(0x7f9f300cd5a0 legacy=0x7f9f30160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300cdb10 con 0x7f9f300d0450 2023-11-08T08:19:06.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9f280019f0 con 0x7f9f300d0450 2023-11-08T08:19:06.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 >> v1:172.21.15.116:6789/0 conn(0x7f9f300d0450 legacy=0x7f9f30160a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f18003a00 0x7f9f18003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18004850 0x7f9f18004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f18005200 0x7f9f1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3006b420 con 0x7f9f18003a00 2023-11-08T08:19:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1800c080 con 0x7f9f18004850 2023-11-08T08:19:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1800c240 con 0x7f9f18005200 2023-11-08T08:19:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f18003a00 0x7f9f18003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2e7fc700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18004850 0x7f9f18004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2f7fe700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f18005200 0x7f9f1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 -- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f18005200 msgr2=0x7f9f1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f18005200 0x7f9f1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 -- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18004850 msgr2=0x7f9f18004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18004850 0x7f9f18004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2effd700 1 -- 172.21.15.116:0/3382943992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180035b0 con 0x7f9f18003a00 2023-11-08T08:19:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2f7fe700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f18005200 0x7f9f1800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.625+0000 7f9f2f7fe700 1 -- 172.21.15.116:0/3382943992 reap_dead start 2023-11-08T08:19:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f18003a00 0x7f9f18003e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009b60 tx=0x7f9f18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78599866a320469 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2000e6c0 con 0x7f9f18003a00 2023-11-08T08:19:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2000e820 con 0x7f9f18003a00 2023-11-08T08:19:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f2dffb700 1 -- 172.21.15.116:0/3382943992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2000eb90 con 0x7f9f18003a00 2023-11-08T08:19:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 -- 172.21.15.116:0/3382943992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f18003a00 msgr2=0x7f9f18003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f18003a00 0x7f9f18003e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009b60 tx=0x7f9f18005880 comp rx=0 tx=0).stop 2023-11-08T08:19:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 -- 172.21.15.116:0/3382943992 shutdown_connections 2023-11-08T08:19:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3382943992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f18003a00 0x7f9f18003e10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 -- 172.21.15.116:0/3382943992 >> 172.21.15.116:0/3382943992 conn(0x7f9f3006c520 msgr2=0x7f9f3006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 -- 172.21.15.116:0/3382943992 shutdown_connections 2023-11-08T08:19:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.626+0000 7f9f357f6700 1 -- 172.21.15.116:0/3382943992 wait complete. 2023-11-08T08:19:06.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 Processor -- start 2023-11-08T08:19:06.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 -- start start 2023-11-08T08:19:06.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18003a00 0x7f9f301f5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f300c6a40 0x7f9f301f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 0x7f9f30156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300d3d70 con 0x7f9f300b3fd0 2023-11-08T08:19:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300c8710 con 0x7f9f18003a00 2023-11-08T08:19:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f357f6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f301efd50 con 0x7f9f300c6a40 2023-11-08T08:19:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.628+0000 7f9f2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 0x7f9f30156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 0x7f9f30156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47782/0 (socket says 172.21.15.116:47782) 2023-11-08T08:19:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 -- 172.21.15.116:0/3739482156 learned_addr learned my addr 172.21.15.116:0/3739482156 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18003a00 0x7f9f301f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2e7fc700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f300c6a40 0x7f9f301f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 -- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f300c6a40 msgr2=0x7f9f301f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f300c6a40 0x7f9f301f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 -- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18003a00 msgr2=0x7f9f301f5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18003a00 0x7f9f301f5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 -- 172.21.15.116:0/3739482156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300d1b70 con 0x7f9f300b3fd0 2023-11-08T08:19:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f2f7fe700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 0x7f9f30156d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f28002010 tx=0x7f9f2800e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2800ec20 con 0x7f9f300b3fd0 2023-11-08T08:19:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2800ed80 con 0x7f9f300b3fd0 2023-11-08T08:19:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f300d1890 con 0x7f9f300b3fd0 2023-11-08T08:19:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28013640 con 0x7f9f300b3fd0 2023-11-08T08:19:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.629+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3015cad0 con 0x7f9f300b3fd0 2023-11-08T08:19:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.630+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f300743f0 con 0x7f9f300b3fd0 2023-11-08T08:19:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.630+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9f280137a0 con 0x7f9f300b3fd0 2023-11-08T08:19:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.631+0000 7f9f177fe700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f1c0682f0 0x7f9f1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.631+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9f28086310 con 0x7f9f300b3fd0 2023-11-08T08:19:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.631+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f1c0682f0 0x7f9f1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.631+0000 7f9f2effd700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f1c0682f0 0x7f9f1c06a7a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f200090b0 tx=0x7f9f2001f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.635+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f28052b70 con 0x7f9f300b3fd0 2023-11-08T08:19:06.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.818+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f300d2c80 con 0x7f9f300b3fd0 2023-11-08T08:19:06.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.820+0000 7f9f177fe700 1 -- 172.21.15.116:0/3739482156 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f9f28056190 con 0x7f9f300b3fd0 2023-11-08T08:19:06.821 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:06.821 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.821+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f1c0682f0 msgr2=0x7f9f1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.821+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f1c0682f0 0x7f9f1c06a7a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f200090b0 tx=0x7f9f2001f040 comp rx=0 tx=0).stop 2023-11-08T08:19:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.821+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 msgr2=0x7f9f30156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.821+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 0x7f9f30156d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f28002010 tx=0x7f9f2800e440 comp rx=0 tx=0).stop 2023-11-08T08:19:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 shutdown_connections 2023-11-08T08:19:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f300b3fd0 0x7f9f30156d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f1c0682f0 0x7f9f1c06a7a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f18003a00 0x7f9f301f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 --2- 172.21.15.116:0/3739482156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f300c6a40 0x7f9f301f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 >> 172.21.15.116:0/3739482156 conn(0x7f9f3006c520 msgr2=0x7f9f301583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:06.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 shutdown_connections 2023-11-08T08:19:06.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.822+0000 7f9f357f6700 1 -- 172.21.15.116:0/3739482156 wait complete. 2023-11-08T08:19:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:06.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:06.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.960+0000 7f8cfd076700 1 Processor -- start 2023-11-08T08:19:06.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.960+0000 7f8cfd076700 1 -- start start 2023-11-08T08:19:06.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cfd076700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 0x7f8cf80b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cfd076700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8cf81599b0 0x7f8cf815d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cfd076700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf8159ec0 con 0x7f8cf80af0a0 2023-11-08T08:19:06.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cfd076700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf815aa60 con 0x7f8cf81599b0 2023-11-08T08:19:06.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cfd076700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80b1f60 con 0x7f8cf80b1650 2023-11-08T08:19:06.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cf759e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8cf80af0a0 0x7f8cf815a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49142/0 (socket says 172.21.15.116:49142) 2023-11-08T08:19:06.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.961+0000 7f8cf759e700 1 -- 172.21.15.116:0/2912304538 learned_addr learned my addr 172.21.15.116:0/2912304538 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:06.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539891979 0 0) 0x7f8cf8159ec0 con 0x7f8cf80af0a0 2023-11-08T08:19:06.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce00035b0 con 0x7f8cf80af0a0 2023-11-08T08:19:06.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 0x7f8cf80b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf659c700 1 --2- 172.21.15.116:0/2912304538 >> v2:172.21.15.129:3300/0 conn(0x7f8cf81599b0 0x7f8cf815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919969258 0 0) 0x7f8ce00035b0 con 0x7f8cf80af0a0 2023-11-08T08:19:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 msgr2=0x7f8cf80b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 0x7f8cf80b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 >> v2:172.21.15.129:3300/0 conn(0x7f8cf81599b0 msgr2=0x7f8cf815d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 --2- 172.21.15.116:0/2912304538 >> v2:172.21.15.129:3300/0 conn(0x7f8cf81599b0 0x7f8cf815d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf815d8a0 con 0x7f8cf80af0a0 2023-11-08T08:19:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f8cf00019f0 con 0x7f8cf80af0a0 2023-11-08T08:19:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 >> v1:172.21.15.116:6789/0 conn(0x7f8cf80af0a0 legacy=0x7f8cf815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0003a00 0x7f8ce0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ce0004430 0x7f8ce00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 0x7f8ce0004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf815aa60 con 0x7f8ce0003a00 2023-11-08T08:19:06.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80b1f60 con 0x7f8ce0004430 2023-11-08T08:19:06.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce000ba10 con 0x7f8cf80b1650 2023-11-08T08:19:06.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0003a00 0x7f8ce0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf659c700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ce0004430 0x7f8ce00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf759e700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 0x7f8ce0004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 -- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 msgr2=0x7f8ce0004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8cf80b1650 0x7f8ce0004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 -- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ce0004430 msgr2=0x7f8ce00048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:06.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ce0004430 0x7f8ce00048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 -- 172.21.15.116:0/2912304538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce00035b0 con 0x7f8ce0003a00 2023-11-08T08:19:06.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf659c700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ce0004430 0x7f8ce00048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:06.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0003a00 0x7f8ce0003e10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009b60 tx=0x7f8ce0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25375d3f96b91c54 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec00e6c0 con 0x7f8ce0003a00 2023-11-08T08:19:06.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec00e820 con 0x7f8ce0003a00 2023-11-08T08:19:06.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.962+0000 7f8cf5d9b700 1 -- 172.21.15.116:0/2912304538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec00eb90 con 0x7f8ce0003a00 2023-11-08T08:19:06.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cfd076700 1 -- 172.21.15.116:0/2912304538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0003a00 msgr2=0x7f8ce0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2912304538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0003a00 0x7f8ce0003e10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009b60 tx=0x7f8ce0005880 comp rx=0 tx=0).stop 2023-11-08T08:19:06.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cf759e700 1 -- 172.21.15.116:0/2912304538 reap_dead start 2023-11-08T08:19:06.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cfd076700 1 -- 172.21.15.116:0/2912304538 shutdown_connections 2023-11-08T08:19:06.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cfd076700 1 -- 172.21.15.116:0/2912304538 >> 172.21.15.116:0/2912304538 conn(0x7f8cf806c010 msgr2=0x7f8cf80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:06.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cfd076700 1 -- 172.21.15.116:0/2912304538 shutdown_connections 2023-11-08T08:19:06.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.963+0000 7f8cfd076700 1 -- 172.21.15.116:0/2912304538 wait complete. 2023-11-08T08:19:06.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.964+0000 7f8cfd076700 1 Processor -- start 2023-11-08T08:19:06.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 -- start start 2023-11-08T08:19:06.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ce0003a00 0x7f8cf81540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf8154b00 0x7f8cf815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 0x7f8cf815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81441d0 con 0x7f8cf80c2b50 2023-11-08T08:19:06.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81550a0 con 0x7f8cf8154b00 2023-11-08T08:19:06.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cfd076700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8150640 con 0x7f8ce0003a00 2023-11-08T08:19:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cf759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 0x7f8cf815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cf759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 0x7f8cf815b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47810/0 (socket says 172.21.15.116:47810) 2023-11-08T08:19:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.965+0000 7f8cf759e700 1 -- 172.21.15.116:0/2693667656 learned_addr learned my addr 172.21.15.116:0/2693667656 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ce0003a00 0x7f8cf81540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf659c700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf8154b00 0x7f8cf815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf759e700 1 -- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ce0003a00 msgr2=0x7f8cf81540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf759e700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ce0003a00 0x7f8cf81540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf759e700 1 -- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf8154b00 msgr2=0x7f8cf815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf759e700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf8154b00 0x7f8cf815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:06.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf759e700 1 -- 172.21.15.116:0/2693667656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf815de60 con 0x7f8cf80c2b50 2023-11-08T08:19:06.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ce0003a00 0x7f8cf81540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:06.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cf759e700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 0x7f8cf815b5e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80b2620 tx=0x7f8cf00073e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0012650 con 0x7f8cf80c2b50 2023-11-08T08:19:06.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf00127b0 con 0x7f8cf80c2b50 2023-11-08T08:19:06.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf815db80 con 0x7f8cf80c2b50 2023-11-08T08:19:06.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0012b60 con 0x7f8cf80c2b50 2023-11-08T08:19:06.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.966+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf815d590 con 0x7f8cf80c2b50 2023-11-08T08:19:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.967+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf80743f0 con 0x7f8cf80c2b50 2023-11-08T08:19:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.967+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8cf0012cc0 con 0x7f8cf80c2b50 2023-11-08T08:19:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.968+0000 7f8cdf7fe700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8ce40682f0 0x7f8ce406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.968+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8ce40682f0 0x7f8ce406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.968+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8cf00883f0 con 0x7f8cf80c2b50 2023-11-08T08:19:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.968+0000 7f8cf6d9d700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8ce40682f0 0x7f8ce406a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0090b0 tx=0x7f8cec008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:06.972+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8cf0054c50 con 0x7f8cf80c2b50 2023-11-08T08:19:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.150+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8cf80c34a0 con 0x7f8cf80c2b50 2023-11-08T08:19:07.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.151+0000 7f8cdf7fe700 1 -- 172.21.15.116:0/2693667656 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8cf0058270 con 0x7f8cf80c2b50 2023-11-08T08:19:07.152 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8ce40682f0 msgr2=0x7f8ce406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8ce40682f0 0x7f8ce406a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0090b0 tx=0x7f8cec008030 comp rx=0 tx=0).stop 2023-11-08T08:19:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 msgr2=0x7f8cf815b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 0x7f8cf815b5e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80b2620 tx=0x7f8cf00073e0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 shutdown_connections 2023-11-08T08:19:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8cf80c2b50 0x7f8cf815b5e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ce0003a00 0x7f8cf81540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8ce40682f0 0x7f8ce406a7a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 --2- 172.21.15.116:0/2693667656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf8154b00 0x7f8cf815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.152+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 >> 172.21.15.116:0/2693667656 conn(0x7f8cf806c010 msgr2=0x7f8cf8159120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.153+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 shutdown_connections 2023-11-08T08:19:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.153+0000 7f8cfd076700 1 -- 172.21.15.116:0/2693667656 wait complete. 2023-11-08T08:19:07.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:07.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.287+0000 7f50c5f0e700 1 Processor -- start 2023-11-08T08:19:07.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.287+0000 7f50c5f0e700 1 -- start start 2023-11-08T08:19:07.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50c5f0e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50c01599b0 0x7f50c015d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50c5f0e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c015a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50c5f0e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c0159ec0 con 0x7f50c00b1650 2023-11-08T08:19:07.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50c5f0e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c015d8a0 con 0x7f50c01599b0 2023-11-08T08:19:07.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50c5f0e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c015db80 con 0x7f50c00af0a0 2023-11-08T08:19:07.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50beffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50c01599b0 0x7f50c015d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50bf7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f50c00b1650 0x7f50c00b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49150/0 (socket says 172.21.15.116:49150) 2023-11-08T08:19:07.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50bf7fe700 1 -- 172.21.15.116:0/1315876045 learned_addr learned my addr 172.21.15.116:0/1315876045 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:07.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50bffff700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50beffd700 1 -- 172.21.15.116:0/1315876045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 msgr2=0x7f50c015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:07.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50beffd700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50beffd700 1 -- 172.21.15.116:0/1315876045 >> v1:172.21.15.116:6789/0 conn(0x7f50c00b1650 legacy=0x7f50c00b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50beffd700 1 -- 172.21.15.116:0/1315876045 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c015de60 con 0x7f50c01599b0 2023-11-08T08:19:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50be7fc700 1 -- 172.21.15.116:0/1315876045 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860735895 0 0) 0x7f50c0159ec0 con 0x7f50c00b1650 2023-11-08T08:19:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50bffff700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50beffd700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c01599b0 0x7f50c015d050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009840 tx=0x7f50b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f23536f53e9f763 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.288+0000 7f50be7fc700 1 -- 172.21.15.116:0/1315876045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0011830 con 0x7f50c01599b0 2023-11-08T08:19:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50be7fc700 1 -- 172.21.15.116:0/1315876045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0011990 con 0x7f50c01599b0 2023-11-08T08:19:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50be7fc700 1 -- 172.21.15.116:0/1315876045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0011d20 con 0x7f50c01599b0 2023-11-08T08:19:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 -- 172.21.15.116:0/1315876045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c01599b0 msgr2=0x7f50c015d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c01599b0 0x7f50c015d050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009840 tx=0x7f50b0009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:07.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 -- 172.21.15.116:0/1315876045 shutdown_connections 2023-11-08T08:19:07.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/1315876045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c01599b0 0x7f50c015d050 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 -- 172.21.15.116:0/1315876045 >> 172.21.15.116:0/1315876045 conn(0x7f50c006c010 msgr2=0x7f50c00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:07.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 -- 172.21.15.116:0/1315876045 shutdown_connections 2023-11-08T08:19:07.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.289+0000 7f50c5f0e700 1 -- 172.21.15.116:0/1315876045 wait complete. 2023-11-08T08:19:07.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.291+0000 7f50c5f0e700 1 Processor -- start 2023-11-08T08:19:07.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.291+0000 7f50c5f0e700 1 -- start start 2023-11-08T08:19:07.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c01460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50c00b1650 0x7f50c00b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 0x7f50c0153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c0146610 con 0x7f50c0156860 2023-11-08T08:19:07.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00b3bf0 con 0x7f50c00b1650 2023-11-08T08:19:07.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00b3ed0 con 0x7f50c00af0a0 2023-11-08T08:19:07.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 0x7f50c0153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 0x7f50c0153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47822/0 (socket says 172.21.15.116:47822) 2023-11-08T08:19:07.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 -- 172.21.15.116:0/3892872341 learned_addr learned my addr 172.21.15.116:0/3892872341 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50beffd700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50c00b1650 0x7f50c00b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bf7fe700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c01460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 -- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 msgr2=0x7f50c01460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c01460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 -- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50c00b1650 msgr2=0x7f50c00b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50c00b1650 0x7f50c00b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 -- 172.21.15.116:0/3892872341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00094a0 con 0x7f50c0156860 2023-11-08T08:19:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50bffff700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 0x7f50c0153d10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f50b800ce60 tx=0x7f50b800e330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8011070 con 0x7f50c0156860 2023-11-08T08:19:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b800c2e0 con 0x7f50c0156860 2023-11-08T08:19:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c015a890 con 0x7f50c0156860 2023-11-08T08:19:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b800eaf0 con 0x7f50c0156860 2023-11-08T08:19:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.292+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c015d590 con 0x7f50c0156860 2023-11-08T08:19:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.293+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c015aa20 con 0x7f50c0156860 2023-11-08T08:19:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.294+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50b800ec50 con 0x7f50c0156860 2023-11-08T08:19:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.294+0000 7f50b7fff700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50a00682f0 0x7f50a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.294+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f50b8087cd0 con 0x7f50c0156860 2023-11-08T08:19:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.294+0000 7f50bf7fe700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50a00682f0 0x7f50a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.295+0000 7f50bf7fe700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50a00682f0 0x7f50a006a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000f60 tx=0x7f50a8007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.298+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50b8054530 con 0x7f50c0156860 2023-11-08T08:19:07.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.478+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50c00743f0 con 0x7f50c0156860 2023-11-08T08:19:07.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.478+0000 7f50b7fff700 1 -- 172.21.15.116:0/3892872341 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f50b8057b50 con 0x7f50c0156860 2023-11-08T08:19:07.480 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:07.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50a00682f0 msgr2=0x7f50a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50a00682f0 0x7f50a006a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000f60 tx=0x7f50a8007e70 comp rx=0 tx=0).stop 2023-11-08T08:19:07.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 msgr2=0x7f50c0153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 0x7f50c0153d10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f50b800ce60 tx=0x7f50b800e330 comp rx=0 tx=0).stop 2023-11-08T08:19:07.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 shutdown_connections 2023-11-08T08:19:07.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0156860 0x7f50c0153d10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50c00af0a0 0x7f50c01460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50a00682f0 0x7f50a006a7a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 --2- 172.21.15.116:0/3892872341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50c00b1650 0x7f50c00b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 >> 172.21.15.116:0/3892872341 conn(0x7f50c006c010 msgr2=0x7f50c00cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 shutdown_connections 2023-11-08T08:19:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.480+0000 7f50c5f0e700 1 -- 172.21.15.116:0/3892872341 wait complete. 2023-11-08T08:19:07.493 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:07.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.615+0000 7f4296593700 1 Processor -- start 2023-11-08T08:19:07.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.615+0000 7f4296593700 1 -- start start 2023-11-08T08:19:07.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4296593700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f429015c810 0x7f4290160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4296593700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f429006a270 0x7f429006a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4296593700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429006abf0 con 0x7f42900718e0 2023-11-08T08:19:07.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4296593700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429015cbe0 con 0x7f429015c810 2023-11-08T08:19:07.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4296593700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42901607a0 con 0x7f429006a270 2023-11-08T08:19:07.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f429015c810 0x7f4290160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428f7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f42900718e0 0x7f429006c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49166/0 (socket says 172.21.15.116:49166) 2023-11-08T08:19:07.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428f7fe700 1 -- 172.21.15.116:0/3559608891 learned_addr learned my addr 172.21.15.116:0/3559608891 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:07.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4294b30700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f429006a270 0x7f429006a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428ffff700 1 -- 172.21.15.116:0/3559608891 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f429006a270 msgr2=0x7f429006a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:07.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428ffff700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f429006a270 0x7f429006a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428ffff700 1 -- 172.21.15.116:0/3559608891 >> v1:172.21.15.116:6789/0 conn(0x7f42900718e0 legacy=0x7f429006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428ffff700 1 -- 172.21.15.116:0/3559608891 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4290160a50 con 0x7f429015c810 2023-11-08T08:19:07.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428effd700 1 -- 172.21.15.116:0/3559608891 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670613576 0 0) 0x7f429006abf0 con 0x7f42900718e0 2023-11-08T08:19:07.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f4294b30700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f429006a270 0x7f429006a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:07.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428ffff700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429015c810 0x7f4290160260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4280009840 tx=0x7f4280009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956a68a0a47df5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428effd700 1 -- 172.21.15.116:0/3559608891 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280011830 con 0x7f429015c810 2023-11-08T08:19:07.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428effd700 1 -- 172.21.15.116:0/3559608891 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4280011990 con 0x7f429015c810 2023-11-08T08:19:07.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.616+0000 7f428effd700 1 -- 172.21.15.116:0/3559608891 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280011d20 con 0x7f429015c810 2023-11-08T08:19:07.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 -- 172.21.15.116:0/3559608891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429015c810 msgr2=0x7f4290160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429015c810 0x7f4290160260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4280009840 tx=0x7f4280009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:07.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 -- 172.21.15.116:0/3559608891 shutdown_connections 2023-11-08T08:19:07.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429015c810 0x7f4290160260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 --2- 172.21.15.116:0/3559608891 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f429006a270 0x7f429006a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 -- 172.21.15.116:0/3559608891 >> 172.21.15.116:0/3559608891 conn(0x7f429006c010 msgr2=0x7f42900cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 -- 172.21.15.116:0/3559608891 shutdown_connections 2023-11-08T08:19:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.617+0000 7f4296593700 1 -- 172.21.15.116:0/3559608891 wait complete. 2023-11-08T08:19:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.618+0000 7f4296593700 1 Processor -- start 2023-11-08T08:19:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.618+0000 7f4296593700 1 -- start start 2023-11-08T08:19:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4296593700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 0x7f429015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4296593700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f42900718e0 0x7f429015d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4296593700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f429015c810 0x7f429015f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4296593700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429015b220 con 0x7f429006a270 2023-11-08T08:19:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4296593700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429015a610 con 0x7f429015c810 2023-11-08T08:19:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4296593700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429015a8f0 con 0x7f42900718e0 2023-11-08T08:19:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 0x7f429015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 0x7f429015cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47830/0 (socket says 172.21.15.116:47830) 2023-11-08T08:19:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 -- 172.21.15.116:0/3566329036 learned_addr learned my addr 172.21.15.116:0/3566329036 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428f7fe700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f42900718e0 0x7f429015d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4294b30700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f429015c810 0x7f429015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 -- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f42900718e0 msgr2=0x7f429015d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f42900718e0 0x7f429015d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 -- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f429015c810 msgr2=0x7f429015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f429015c810 0x7f429015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 -- 172.21.15.116:0/3566329036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800094a0 con 0x7f429006a270 2023-11-08T08:19:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f4294b30700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f429015c810 0x7f429015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.619+0000 7f428ffff700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 0x7f429015cfc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4280014040 tx=0x7f4280004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.620+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42800045f0 con 0x7f429006a270 2023-11-08T08:19:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.620+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428001e070 con 0x7f429006a270 2023-11-08T08:19:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.620+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4290161010 con 0x7f429006a270 2023-11-08T08:19:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.620+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280003ae0 con 0x7f429006a270 2023-11-08T08:19:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.620+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4290160d30 con 0x7f429006a270 2023-11-08T08:19:07.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.621+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42900ceb20 con 0x7f429006a270 2023-11-08T08:19:07.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.621+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f428000f050 con 0x7f429006a270 2023-11-08T08:19:07.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.622+0000 7f428cff9700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f427c0682f0 0x7f427c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.622+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4280087260 con 0x7f429006a270 2023-11-08T08:19:07.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.622+0000 7f428f7fe700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f427c0682f0 0x7f427c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.622+0000 7f428f7fe700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f427c0682f0 0x7f427c06a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4284000f60 tx=0x7f4284007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.626+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4280053b60 con 0x7f429006a270 2023-11-08T08:19:07.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.802+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42900743f0 con 0x7f429006a270 2023-11-08T08:19:07.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.803+0000 7f428cff9700 1 -- 172.21.15.116:0/3566329036 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4280057180 con 0x7f429006a270 2023-11-08T08:19:07.804 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:07.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f427c0682f0 msgr2=0x7f427c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f427c0682f0 0x7f427c06a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4284000f60 tx=0x7f4284007be0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 msgr2=0x7f429015cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 0x7f429015cfc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4280014040 tx=0x7f4280004750 comp rx=0 tx=0).stop 2023-11-08T08:19:07.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 shutdown_connections 2023-11-08T08:19:07.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f427c0682f0 0x7f427c06a7a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f429015c810 0x7f429015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f429006a270 0x7f429015cfc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 --2- 172.21.15.116:0/3566329036 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f42900718e0 0x7f429015d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 >> 172.21.15.116:0/3566329036 conn(0x7f429006c010 msgr2=0x7f42900caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 shutdown_connections 2023-11-08T08:19:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.805+0000 7f4296593700 1 -- 172.21.15.116:0/3566329036 wait complete. 2023-11-08T08:19:07.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:07.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.939+0000 7f8bcd5fa700 1 Processor -- start 2023-11-08T08:19:07.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.939+0000 7f8bcd5fa700 1 -- start start 2023-11-08T08:19:07.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bcd5fa700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc815a590 0x7f8bc806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bcd5fa700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bc80b50d0 0x7f8bc806c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bcd5fa700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8bc815a960 0x7f8bc806c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bcd5fa700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80cd480 con 0x7f8bc80b50d0 2023-11-08T08:19:07.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bcd5fa700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80cd760 con 0x7f8bc815a960 2023-11-08T08:19:07.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bcd5fa700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80cda40 con 0x7f8bc815a590 2023-11-08T08:19:07.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bc67fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bc80b50d0 0x7f8bc806c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bc67fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bc80b50d0 0x7f8bc806c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47840/0 (socket says 172.21.15.116:47840) 2023-11-08T08:19:07.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.940+0000 7f8bc67fc700 1 -- 172.21.15.116:0/1545927289 learned_addr learned my addr 172.21.15.116:0/1545927289 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:07.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc77fe700 1 --2- 172.21.15.116:0/1545927289 >> v2:172.21.15.129:3300/0 conn(0x7f8bc815a960 0x7f8bc806c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc6ffd700 1 --2- 172.21.15.116:0/1545927289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc815a590 0x7f8bc806b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc67fc700 1 -- 172.21.15.116:0/1545927289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc815a590 msgr2=0x7f8bc806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:07.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc67fc700 1 --2- 172.21.15.116:0/1545927289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc815a590 0x7f8bc806b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc67fc700 1 -- 172.21.15.116:0/1545927289 >> v2:172.21.15.129:3300/0 conn(0x7f8bc815a960 msgr2=0x7f8bc806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:07.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc67fc700 1 --2- 172.21.15.116:0/1545927289 >> v2:172.21.15.129:3300/0 conn(0x7f8bc815a960 0x7f8bc806c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc67fc700 1 -- 172.21.15.116:0/1545927289 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80cdd20 con 0x7f8bc80b50d0 2023-11-08T08:19:07.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc67fc700 1 --2- 172.21.15.116:0/1545927289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc80b50d0 0x7f8bc806c440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80cbe10 tx=0x7f8bb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11766272a0996029 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc5ffb700 1 -- 172.21.15.116:0/1545927289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb800b870 con 0x7f8bc80b50d0 2023-11-08T08:19:07.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc5ffb700 1 -- 172.21.15.116:0/1545927289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb800b9d0 con 0x7f8bc80b50d0 2023-11-08T08:19:07.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bc5ffb700 1 -- 172.21.15.116:0/1545927289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb800bd60 con 0x7f8bc80b50d0 2023-11-08T08:19:07.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/1545927289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc80b50d0 msgr2=0x7f8bc806c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/1545927289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc80b50d0 0x7f8bc806c440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80cbe10 tx=0x7f8bb8009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/1545927289 shutdown_connections 2023-11-08T08:19:07.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/1545927289 >> v2:172.21.15.129:3300/0 conn(0x7f8bc815a960 0x7f8bc806c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/1545927289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc815a590 0x7f8bc806b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/1545927289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc80b50d0 0x7f8bc806c440 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.941+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/1545927289 >> 172.21.15.116:0/1545927289 conn(0x7f8bc80cf400 msgr2=0x7f8bc80cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.942+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/1545927289 shutdown_connections 2023-11-08T08:19:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.942+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/1545927289 wait complete. 2023-11-08T08:19:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.943+0000 7f8bcd5fa700 1 Processor -- start 2023-11-08T08:19:07.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.943+0000 7f8bcd5fa700 1 -- start start 2023-11-08T08:19:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.943+0000 7f8bcd5fa700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc80b50d0 0x7f8bc814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.943+0000 7f8bcd5fa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bc815a590 0x7f8bc814f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.943+0000 7f8bcd5fa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 0x7f8bc80c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bcd5fa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc815c970 con 0x7f8bc815a960 2023-11-08T08:19:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bcd5fa700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc815cc50 con 0x7f8bc815a590 2023-11-08T08:19:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bcd5fa700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc815cf30 con 0x7f8bc80b50d0 2023-11-08T08:19:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 0x7f8bc80c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 0x7f8bc80c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47842/0 (socket says 172.21.15.116:47842) 2023-11-08T08:19:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 -- 172.21.15.116:0/723225146 learned_addr learned my addr 172.21.15.116:0/723225146 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc6ffd700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc80b50d0 0x7f8bc814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc67fc700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bc815a590 0x7f8bc814f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 -- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc80b50d0 msgr2=0x7f8bc814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc80b50d0 0x7f8bc814f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 -- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bc815a590 msgr2=0x7f8bc814f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:07.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bc815a590 0x7f8bc814f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:07.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 -- 172.21.15.116:0/723225146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8009550 con 0x7f8bc815a960 2023-11-08T08:19:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bc77fe700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 0x7f8bc80c8330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0007e60 tx=0x7f8bc000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc000ed30 con 0x7f8bc815a960 2023-11-08T08:19:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc0014420 con 0x7f8bc815a960 2023-11-08T08:19:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc00147d0 con 0x7f8bc815a960 2023-11-08T08:19:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc806baf0 con 0x7f8bc815a960 2023-11-08T08:19:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.944+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc80ce000 con 0x7f8bc815a960 2023-11-08T08:19:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.945+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc815e280 con 0x7f8bc815a960 2023-11-08T08:19:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.946+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8bc0012050 con 0x7f8bc815a960 2023-11-08T08:19:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.946+0000 7f8baf7fe700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8bb40682f0 0x7f8bb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.946+0000 7f8bc6ffd700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8bb40682f0 0x7f8bb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.946+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8bc0086110 con 0x7f8bc815a960 2023-11-08T08:19:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.947+0000 7f8bc6ffd700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8bb40682f0 0x7f8bb406a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc006b20 tx=0x7f8bbc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:07.950+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8bc0052970 con 0x7f8bc815a960 2023-11-08T08:19:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.127+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8bc80743f0 con 0x7f8bc815a960 2023-11-08T08:19:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.127+0000 7f8baf7fe700 1 -- 172.21.15.116:0/723225146 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8bc0055f90 con 0x7f8bc815a960 2023-11-08T08:19:08.129 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8bb40682f0 msgr2=0x7f8bb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8bb40682f0 0x7f8bb406a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc006b20 tx=0x7f8bbc008040 comp rx=0 tx=0).stop 2023-11-08T08:19:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 msgr2=0x7f8bc80c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 0x7f8bc80c8330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0007e60 tx=0x7f8bc000e3c0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 shutdown_connections 2023-11-08T08:19:08.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bc815a960 0x7f8bc80c8330 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8bc80b50d0 0x7f8bc814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8bb40682f0 0x7f8bb406a7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 --2- 172.21.15.116:0/723225146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bc815a590 0x7f8bc814f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 >> 172.21.15.116:0/723225146 conn(0x7f8bc80cf400 msgr2=0x7f8bc806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 shutdown_connections 2023-11-08T08:19:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.129+0000 7f8bcd5fa700 1 -- 172.21.15.116:0/723225146 wait complete. 2023-11-08T08:19:08.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.264+0000 7f86f4925700 1 Processor -- start 2023-11-08T08:19:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.264+0000 7f86f4925700 1 -- start start 2023-11-08T08:19:08.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86f4925700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00ce040 0x7f86f015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86f4925700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f013fc90 con 0x7f86f00b1770 2023-11-08T08:19:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86f4925700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f013f480 con 0x7f86f0142db0 2023-11-08T08:19:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86f4925700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f013f6d0 con 0x7f86f00ce040 2023-11-08T08:19:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86eed9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f86f00b1770 0x7f86f013cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49182/0 (socket says 172.21.15.116:49182) 2023-11-08T08:19:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86eed9d700 1 -- 172.21.15.116:0/738187401 learned_addr learned my addr 172.21.15.116:0/738187401 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ee59c700 1 --2- 172.21.15.116:0/738187401 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00ce040 0x7f86f015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186222145 0 0) 0x7f86f013fc90 con 0x7f86f00b1770 2023-11-08T08:19:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d80057c0 con 0x7f86f00b1770 2023-11-08T08:19:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308420385 0 0) 0x7f86f013f480 con 0x7f86f0142db0 2023-11-08T08:19:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f013fc90 con 0x7f86f0142db0 2023-11-08T08:19:08.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234477401 0 0) 0x7f86d80057c0 con 0x7f86f00b1770 2023-11-08T08:19:08.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ee59c700 1 -- 172.21.15.116:0/738187401 >> v1:172.21.15.129:6789/0 conn(0x7f86f0142db0 legacy=0x7f86f013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ee59c700 1 -- 172.21.15.116:0/738187401 >> v1:172.21.15.116:6789/0 conn(0x7f86f00b1770 legacy=0x7f86f013cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.265+0000 7f86ee59c700 1 -- 172.21.15.116:0/738187401 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f013f9b0 con 0x7f86f00ce040 2023-11-08T08:19:08.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86ee59c700 1 --2- 172.21.15.116:0/738187401 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00ce040 0x7f86f015d900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009870 tx=0x7f86e0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf35b3f3c0b542f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e80019f0 con 0x7f86f00ce040 2023-11-08T08:19:08.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e0004030 con 0x7f86f00ce040 2023-11-08T08:19:08.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86ed59a700 1 -- 172.21.15.116:0/738187401 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e00043a0 con 0x7f86f00ce040 2023-11-08T08:19:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 -- 172.21.15.116:0/738187401 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00ce040 msgr2=0x7f86f015d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 --2- 172.21.15.116:0/738187401 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00ce040 0x7f86f015d900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009870 tx=0x7f86e0009b80 comp rx=0 tx=0).stop 2023-11-08T08:19:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 -- 172.21.15.116:0/738187401 shutdown_connections 2023-11-08T08:19:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 --2- 172.21.15.116:0/738187401 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00ce040 0x7f86f015d900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 -- 172.21.15.116:0/738187401 >> 172.21.15.116:0/738187401 conn(0x7f86f006c010 msgr2=0x7f86f00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 -- 172.21.15.116:0/738187401 shutdown_connections 2023-11-08T08:19:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.266+0000 7f86f4925700 1 -- 172.21.15.116:0/738187401 wait complete. 2023-11-08T08:19:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 Processor -- start 2023-11-08T08:19:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 -- start start 2023-11-08T08:19:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00b1770 0x7f86f01f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 0x7f86f00cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f0142db0 0x7f86f00cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f01515a0 con 0x7f86f00ce040 2023-11-08T08:19:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00c6670 con 0x7f86f0142db0 2023-11-08T08:19:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86f4925700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00c37f0 con 0x7f86f00b1770 2023-11-08T08:19:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86edd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 0x7f86f00cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86edd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 0x7f86f00cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47858/0 (socket says 172.21.15.116:47858) 2023-11-08T08:19:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.268+0000 7f86edd9b700 1 -- 172.21.15.116:0/4285447052 learned_addr learned my addr 172.21.15.116:0/4285447052 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86ee59c700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00b1770 0x7f86f01f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86eed9d700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f0142db0 0x7f86f00cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86edd9b700 1 -- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00b1770 msgr2=0x7f86f01f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86edd9b700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00b1770 0x7f86f01f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86edd9b700 1 -- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f0142db0 msgr2=0x7f86f00cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86edd9b700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f0142db0 0x7f86f00cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86edd9b700 1 -- 172.21.15.116:0/4285447052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e00094a0 con 0x7f86f00ce040 2023-11-08T08:19:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86eed9d700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f0142db0 0x7f86f00cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86edd9b700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 0x7f86f00cc770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f86e400c500 tx=0x7f86e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e40052d0 con 0x7f86f00ce040 2023-11-08T08:19:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e4005430 con 0x7f86f00ce040 2023-11-08T08:19:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f015d3e0 con 0x7f86f00ce040 2023-11-08T08:19:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e40038c0 con 0x7f86f00ce040 2023-11-08T08:19:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.269+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f015d100 con 0x7f86f00ce040 2023-11-08T08:19:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.270+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f00ce410 con 0x7f86f00ce040 2023-11-08T08:19:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.271+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f86e4019030 con 0x7f86f00ce040 2023-11-08T08:19:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.271+0000 7f86deffd700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86d00682f0 0x7f86d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.271+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f86e4085130 con 0x7f86f00ce040 2023-11-08T08:19:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.271+0000 7f86ee59c700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86d00682f0 0x7f86d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.271+0000 7f86ee59c700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86d00682f0 0x7f86d006a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86e0000c00 tx=0x7f86e0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.275+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f86e40518a0 con 0x7f86f00ce040 2023-11-08T08:19:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.456+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86f00743f0 con 0x7f86f00ce040 2023-11-08T08:19:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.458+0000 7f86deffd700 1 -- 172.21.15.116:0/4285447052 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f86e4054ec0 con 0x7f86f00ce040 2023-11-08T08:19:08.459 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:08.459 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86d00682f0 msgr2=0x7f86d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86d00682f0 0x7f86d006a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86e0000c00 tx=0x7f86e0008030 comp rx=0 tx=0).stop 2023-11-08T08:19:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 msgr2=0x7f86f00cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 0x7f86f00cc770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f86e400c500 tx=0x7f86e400c8c0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 shutdown_connections 2023-11-08T08:19:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86d00682f0 0x7f86d006a7a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f0142db0 0x7f86f00cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86f00b1770 0x7f86f01f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 --2- 172.21.15.116:0/4285447052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f00ce040 0x7f86f00cc770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.459+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 >> 172.21.15.116:0/4285447052 conn(0x7f86f006c010 msgr2=0x7f86f01586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.460+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 shutdown_connections 2023-11-08T08:19:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.460+0000 7f86f4925700 1 -- 172.21.15.116:0/4285447052 wait complete. 2023-11-08T08:19:08.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:08.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.595+0000 7f5804d30700 1 Processor -- start 2023-11-08T08:19:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.595+0000 7f5804d30700 1 -- start start 2023-11-08T08:19:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.596+0000 7f5804d30700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 0x7f58000b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.596+0000 7f5804d30700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5800159ec0 con 0x7f58001599b0 2023-11-08T08:19:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.596+0000 7f5804d30700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580015aa60 con 0x7f58000af0a0 2023-11-08T08:19:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.596+0000 7f5804d30700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58000b1f60 con 0x7f58000b1650 2023-11-08T08:19:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fdd9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f58001599b0 0x7f580015d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49186/0 (socket says 172.21.15.116:49186) 2023-11-08T08:19:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fdd9b700 1 -- 172.21.15.116:0/1789770260 learned_addr learned my addr 172.21.15.116:0/1789770260 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fe59c700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 0x7f58000b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351068892 0 0) 0x7f5800159ec0 con 0x7f58001599b0 2023-11-08T08:19:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e80035b0 con 0x7f58001599b0 2023-11-08T08:19:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4066375166 0 0) 0x7f57e80035b0 con 0x7f58001599b0 2023-11-08T08:19:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 msgr2=0x7f58000b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 0x7f58000b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 >> v1:172.21.15.129:6789/0 conn(0x7f58000af0a0 legacy=0x7f580015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580015d8a0 con 0x7f58001599b0 2023-11-08T08:19:08.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f57f0001670 con 0x7f58001599b0 2023-11-08T08:19:08.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 >> v1:172.21.15.116:6789/0 conn(0x7f58001599b0 legacy=0x7f580015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8003a30 0x7f57e8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e8004850 0x7f57e8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 0x7f57e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000b1f60 con 0x7f57e8004850 2023-11-08T08:19:08.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e800bac0 con 0x7f57e8003a30 2023-11-08T08:19:08.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e800bc50 con 0x7f58000b1650 2023-11-08T08:19:08.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fdd9b700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e8004850 0x7f57e8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fe59c700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8003a30 0x7f57e8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.597+0000 7f57fed9d700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 0x7f57e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fdd9b700 1 -- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 msgr2=0x7f57e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:08.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fdd9b700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b1650 0x7f57e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fdd9b700 1 -- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8003a30 msgr2=0x7f57e8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:08.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fdd9b700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8003a30 0x7f57e8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fdd9b700 1 -- 172.21.15.116:0/1789770260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e80035b0 con 0x7f57e8004850 2023-11-08T08:19:08.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fdd9b700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e8004850 0x7f57e8004cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57f000c210 tx=0x7f57e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa53e5d34adf7c8c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0007c10 con 0x7f57e8004850 2023-11-08T08:19:08.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f000cdd0 con 0x7f57e8004850 2023-11-08T08:19:08.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fd59a700 1 -- 172.21.15.116:0/1789770260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0005ba0 con 0x7f57e8004850 2023-11-08T08:19:08.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f5804d30700 1 -- 172.21.15.116:0/1789770260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e8004850 msgr2=0x7f57e8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f5804d30700 1 --2- 172.21.15.116:0/1789770260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e8004850 0x7f57e8004cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57f000c210 tx=0x7f57e8005880 comp rx=0 tx=0).stop 2023-11-08T08:19:08.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f57fed9d700 1 -- 172.21.15.116:0/1789770260 reap_dead start 2023-11-08T08:19:08.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f5804d30700 1 -- 172.21.15.116:0/1789770260 shutdown_connections 2023-11-08T08:19:08.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f5804d30700 1 -- 172.21.15.116:0/1789770260 >> 172.21.15.116:0/1789770260 conn(0x7f580006c010 msgr2=0x7f58000cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:08.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f5804d30700 1 -- 172.21.15.116:0/1789770260 shutdown_connections 2023-11-08T08:19:08.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.598+0000 7f5804d30700 1 -- 172.21.15.116:0/1789770260 wait complete. 2023-11-08T08:19:08.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 Processor -- start 2023-11-08T08:19:08.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 -- start start 2023-11-08T08:19:08.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8004850 0x7f5800149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b37f0 0x7f58000c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 0x7f580015cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580015d450 con 0x7f580015caa0 2023-11-08T08:19:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58001505d0 con 0x7f57e8004850 2023-11-08T08:19:08.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.600+0000 7f5804d30700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58001495d0 con 0x7f58000b37f0 2023-11-08T08:19:08.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 0x7f580015cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 0x7f580015cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47884/0 (socket says 172.21.15.116:47884) 2023-11-08T08:19:08.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 -- 172.21.15.116:0/872598141 learned_addr learned my addr 172.21.15.116:0/872598141 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fdd9b700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b37f0 0x7f58000c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fe59c700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8004850 0x7f5800149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 -- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b37f0 msgr2=0x7f58000c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b37f0 0x7f58000c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 -- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8004850 msgr2=0x7f5800149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8004850 0x7f5800149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 -- 172.21.15.116:0/872598141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580015de60 con 0x7f580015caa0 2023-11-08T08:19:08.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57fed9d700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 0x7f580015cf10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f58000b2620 tx=0x7f57f800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f8003c10 con 0x7f580015caa0 2023-11-08T08:19:08.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f8003d90 con 0x7f580015caa0 2023-11-08T08:19:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580015db80 con 0x7f580015caa0 2023-11-08T08:19:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800d950 con 0x7f580015caa0 2023-11-08T08:19:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.601+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58000b1a20 con 0x7f580015caa0 2023-11-08T08:19:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.602+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58000743f0 con 0x7f580015caa0 2023-11-08T08:19:08.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.602+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57f800dab0 con 0x7f580015caa0 2023-11-08T08:19:08.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.603+0000 7f57eeffd700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57e00682f0 0x7f57e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.603+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f57f8085ef0 con 0x7f580015caa0 2023-11-08T08:19:08.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.603+0000 7f57fe59c700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57e00682f0 0x7f57e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.603+0000 7f57fe59c700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57e00682f0 0x7f57e006a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009c60 tx=0x7f57f4009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.607+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57f8052750 con 0x7f580015caa0 2023-11-08T08:19:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.787+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f580006e620 con 0x7f580015caa0 2023-11-08T08:19:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.787+0000 7f57eeffd700 1 -- 172.21.15.116:0/872598141 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f57f8055d70 con 0x7f580015caa0 2023-11-08T08:19:08.789 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57e00682f0 msgr2=0x7f57e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57e00682f0 0x7f57e006a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009c60 tx=0x7f57f4009210 comp rx=0 tx=0).stop 2023-11-08T08:19:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 msgr2=0x7f580015cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 0x7f580015cf10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f58000b2620 tx=0x7f57f800d150 comp rx=0 tx=0).stop 2023-11-08T08:19:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 shutdown_connections 2023-11-08T08:19:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f580015caa0 0x7f580015cf10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57e00682f0 0x7f57e006a7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e8004850 0x7f5800149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 --2- 172.21.15.116:0/872598141 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58000b37f0 0x7f58000c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 >> 172.21.15.116:0/872598141 conn(0x7f580006c010 msgr2=0x7f58000b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 shutdown_connections 2023-11-08T08:19:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.789+0000 7f5804d30700 1 -- 172.21.15.116:0/872598141 wait complete. 2023-11-08T08:19:08.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.924+0000 7fcd4599b700 1 Processor -- start 2023-11-08T08:19:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.924+0000 7fcd4599b700 1 -- start start 2023-11-08T08:19:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.925+0000 7fcd4599b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcd4015c8c0 0x7fcd40160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.925+0000 7fcd4599b700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400d2d30 con 0x7fcd400b5dc0 2023-11-08T08:19:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.925+0000 7fcd4599b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400d0b40 con 0x7fcd4006b220 2023-11-08T08:19:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.925+0000 7fcd4599b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd401611f0 con 0x7fcd4015c8c0 2023-11-08T08:19:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3e7fc700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcd400b5dc0 0x7fcd4006ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49190/0 (socket says 172.21.15.116:49190) 2023-11-08T08:19:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3e7fc700 1 -- 172.21.15.116:0/1974358604 learned_addr learned my addr 172.21.15.116:0/1974358604 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3effd700 1 --2- 172.21.15.116:0/1974358604 >> v2:172.21.15.129:3300/0 conn(0x7fcd4015c8c0 0x7fcd40160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877282395 0 0) 0x7fcd30001110 con 0x7fcd400b5dc0 2023-11-08T08:19:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd280035b0 con 0x7fcd400b5dc0 2023-11-08T08:19:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638510601 0 0) 0x7fcd280035b0 con 0x7fcd400b5dc0 2023-11-08T08:19:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 >> v2:172.21.15.129:3300/0 conn(0x7fcd4015c8c0 msgr2=0x7fcd40160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 --2- 172.21.15.116:0/1974358604 >> v2:172.21.15.129:3300/0 conn(0x7fcd4015c8c0 0x7fcd40160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 >> v1:172.21.15.129:6789/0 conn(0x7fcd4006b220 legacy=0x7fcd400d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400d2a50 con 0x7fcd400b5dc0 2023-11-08T08:19:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fcd30001800 con 0x7fcd400b5dc0 2023-11-08T08:19:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 >> v1:172.21.15.116:6789/0 conn(0x7fcd400b5dc0 legacy=0x7fcd4006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd28003a00 0x7fcd28003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd28004850 0x7fcd28004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28005200 0x7fcd2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd401611f0 con 0x7fcd28003a00 2023-11-08T08:19:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800c080 con 0x7fcd28005200 2023-11-08T08:19:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800c240 con 0x7fcd28004850 2023-11-08T08:19:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.926+0000 7fcd3effd700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd28003a00 0x7fcd28003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3effd700 1 -- 172.21.15.116:0/1974358604 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd28004850 msgr2=0x7fcd28004cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:19:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3effd700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd28004850 0x7fcd28004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3effd700 1 -- 172.21.15.116:0/1974358604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28005200 msgr2=0x7fcd2800ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:19:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3effd700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28005200 0x7fcd2800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3effd700 1 -- 172.21.15.116:0/1974358604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd280035b0 con 0x7fcd28003a00 2023-11-08T08:19:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3f7fe700 1 -- 172.21.15.116:0/1974358604 reap_dead start 2023-11-08T08:19:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3effd700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd28003a00 0x7fcd28003e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009b60 tx=0x7fcd28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf600750b5bb6a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3400e6c0 con 0x7fcd28003a00 2023-11-08T08:19:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3400e820 con 0x7fcd28003a00 2023-11-08T08:19:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd3dffb700 1 -- 172.21.15.116:0/1974358604 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3400eb90 con 0x7fcd28003a00 2023-11-08T08:19:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 -- 172.21.15.116:0/1974358604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd28003a00 msgr2=0x7fcd28003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd28003a00 0x7fcd28003e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009b60 tx=0x7fcd28005880 comp rx=0 tx=0).stop 2023-11-08T08:19:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 -- 172.21.15.116:0/1974358604 shutdown_connections 2023-11-08T08:19:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 --2- 172.21.15.116:0/1974358604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd28003a00 0x7fcd28003e10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 -- 172.21.15.116:0/1974358604 >> 172.21.15.116:0/1974358604 conn(0x7fcd4006c520 msgr2=0x7fcd4006c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 -- 172.21.15.116:0/1974358604 shutdown_connections 2023-11-08T08:19:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.927+0000 7fcd4599b700 1 -- 172.21.15.116:0/1974358604 wait complete. 2023-11-08T08:19:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.928+0000 7fcd4599b700 1 Processor -- start 2023-11-08T08:19:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 -- start start 2023-11-08T08:19:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28003a00 0x7fcd400cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 0x7fcd400b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4014e000 0x7fcd4014e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4014e9c0 con 0x7fcd400ccbd0 2023-11-08T08:19:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400cb4f0 con 0x7fcd28003a00 2023-11-08T08:19:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.929+0000 7fcd4599b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400cb7d0 con 0x7fcd4014e000 2023-11-08T08:19:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 0x7fcd400b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 0x7fcd400b2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47890/0 (socket says 172.21.15.116:47890) 2023-11-08T08:19:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 -- 172.21.15.116:0/3945228229 learned_addr learned my addr 172.21.15.116:0/3945228229 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3effd700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28003a00 0x7fcd400cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 -- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4014e000 msgr2=0x7fcd4014e480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:19:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4014e000 0x7fcd4014e480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 -- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28003a00 msgr2=0x7fcd400cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28003a00 0x7fcd400cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 -- 172.21.15.116:0/3945228229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400cda40 con 0x7fcd400ccbd0 2023-11-08T08:19:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3effd700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28003a00 0x7fcd400cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd3e7fc700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 0x7fcd400b2c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcd30000fb0 tx=0x7fcd3000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30004a80 con 0x7fcd400ccbd0 2023-11-08T08:19:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd30004be0 con 0x7fcd400ccbd0 2023-11-08T08:19:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd400cd760 con 0x7fcd400ccbd0 2023-11-08T08:19:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd300070d0 con 0x7fcd400ccbd0 2023-11-08T08:19:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.930+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd400d2700 con 0x7fcd400ccbd0 2023-11-08T08:19:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.931+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd40067d70 con 0x7fcd400ccbd0 2023-11-08T08:19:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.932+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd3001a030 con 0x7fcd400ccbd0 2023-11-08T08:19:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.932+0000 7fcd27fff700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd2c0682f0 0x7fcd2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.932+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcd30084f20 con 0x7fcd400ccbd0 2023-11-08T08:19:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.932+0000 7fcd3effd700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd2c0682f0 0x7fcd2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.932+0000 7fcd3effd700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd2c0682f0 0x7fcd2c06a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcd340090b0 tx=0x7fcd34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:08.936+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd30051780 con 0x7fcd400ccbd0 2023-11-08T08:19:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.113+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd400d2d30 con 0x7fcd400ccbd0 2023-11-08T08:19:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.114+0000 7fcd27fff700 1 -- 172.21.15.116:0/3945228229 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd400d2d30 con 0x7fcd400ccbd0 2023-11-08T08:19:09.115 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd2c0682f0 msgr2=0x7fcd2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd2c0682f0 0x7fcd2c06a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcd340090b0 tx=0x7fcd34008030 comp rx=0 tx=0).stop 2023-11-08T08:19:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 msgr2=0x7fcd400b2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 0x7fcd400b2c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcd30000fb0 tx=0x7fcd3000ce00 comp rx=0 tx=0).stop 2023-11-08T08:19:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 shutdown_connections 2023-11-08T08:19:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4014e000 0x7fcd4014e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd2c0682f0 0x7fcd2c06a7a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd28003a00 0x7fcd400cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 --2- 172.21.15.116:0/3945228229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400ccbd0 0x7fcd400b2c20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 >> 172.21.15.116:0/3945228229 conn(0x7fcd4006c520 msgr2=0x7fcd400b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 shutdown_connections 2023-11-08T08:19:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.116+0000 7fcd4599b700 1 -- 172.21.15.116:0/3945228229 wait complete. 2023-11-08T08:19:09.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.252+0000 7fdc135e9700 1 Processor -- start 2023-11-08T08:19:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.252+0000 7fdc135e9700 1 -- start start 2023-11-08T08:19:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc135e9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc135e9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc0c0c9a50 0x7fdc0c0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc135e9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c06a310 con 0x7fdc0c159710 2023-11-08T08:19:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc135e9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0707c0 con 0x7fdc0c0c9a50 2023-11-08T08:19:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc135e9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0ca410 con 0x7fdc0c070370 2023-11-08T08:19:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc0c0c9a50 0x7fdc0c0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc0c0c9a50 0x7fdc0c0c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47900/0 (socket says 172.21.15.116:47900) 2023-11-08T08:19:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 -- 172.21.15.116:0/3892743991 learned_addr learned my addr 172.21.15.116:0/3892743991 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc10b84700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 -- 172.21.15.116:0/3892743991 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 msgr2=0x7fdc0c0cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 -- 172.21.15.116:0/3892743991 >> v1:172.21.15.129:6789/0 conn(0x7fdc0c159710 legacy=0x7fdc0c0cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 -- 172.21.15.116:0/3892743991 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0ce7e0 con 0x7fdc0c0c9a50 2023-11-08T08:19:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc10b84700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc11b86700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9ed0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009e50 tx=0x7fdc0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed060341976b3ab2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc03fff700 1 -- 172.21.15.116:0/3892743991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400e070 con 0x7fdc0c0c9a50 2023-11-08T08:19:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc03fff700 1 -- 172.21.15.116:0/3892743991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04003a20 con 0x7fdc0c0c9a50 2023-11-08T08:19:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.253+0000 7fdc03fff700 1 -- 172.21.15.116:0/3892743991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04003dd0 con 0x7fdc0c0c9a50 2023-11-08T08:19:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 -- 172.21.15.116:0/3892743991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c0c9a50 msgr2=0x7fdc0c0c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9ed0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009e50 tx=0x7fdc0400b330 comp rx=0 tx=0).stop 2023-11-08T08:19:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 -- 172.21.15.116:0/3892743991 shutdown_connections 2023-11-08T08:19:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9ed0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3892743991 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 -- 172.21.15.116:0/3892743991 >> 172.21.15.116:0/3892743991 conn(0x7fdc0c0cf400 msgr2=0x7fdc0c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 -- 172.21.15.116:0/3892743991 shutdown_connections 2023-11-08T08:19:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.254+0000 7fdc135e9700 1 -- 172.21.15.116:0/3892743991 wait complete. 2023-11-08T08:19:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.255+0000 7fdc135e9700 1 Processor -- start 2023-11-08T08:19:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.255+0000 7fdc135e9700 1 -- start start 2023-11-08T08:19:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc135e9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc135e9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc135e9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 0x7fdc0c157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc135e9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c158c70 con 0x7fdc0c159710 2023-11-08T08:19:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc135e9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0c9700 con 0x7fdc0c0c9a50 2023-11-08T08:19:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc135e9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0c2fd0 con 0x7fdc0c070370 2023-11-08T08:19:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 0x7fdc0c157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 0x7fdc0c157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47910/0 (socket says 172.21.15.116:47910) 2023-11-08T08:19:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 -- 172.21.15.116:0/3531934748 learned_addr learned my addr 172.21.15.116:0/3531934748 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc10b84700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11385700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 -- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 msgr2=0x7fdc0c0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 -- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc0c0c9a50 msgr2=0x7fdc0c0c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 -- 172.21.15.116:0/3531934748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc04009b30 con 0x7fdc0c159710 2023-11-08T08:19:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.256+0000 7fdc11b86700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 0x7fdc0c157c20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c06beb0 tx=0x7fdc04004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.257+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04004440 con 0x7fdc0c159710 2023-11-08T08:19:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.257+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc040045a0 con 0x7fdc0c159710 2023-11-08T08:19:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.257+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04004950 con 0x7fdc0c159710 2023-11-08T08:19:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.257+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c06a890 con 0x7fdc0c159710 2023-11-08T08:19:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.257+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c06a5b0 con 0x7fdc0c159710 2023-11-08T08:19:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.258+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c0ceac0 con 0x7fdc0c159710 2023-11-08T08:19:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.258+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc0402c070 con 0x7fdc0c159710 2023-11-08T08:19:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.258+0000 7fdc01ffb700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdbf00682f0 0x7fdbf006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.258+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdc04087480 con 0x7fdc0c159710 2023-11-08T08:19:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.258+0000 7fdc11385700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdbf00682f0 0x7fdbf006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.259+0000 7fdc11385700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdbf00682f0 0x7fdbf006a7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0b3bf0 tx=0x7fdc08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.263+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc04053c60 con 0x7fdc0c159710 2023-11-08T08:19:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.441+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdc0c0743f0 con 0x7fdc0c159710 2023-11-08T08:19:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.441+0000 7fdc01ffb700 1 -- 172.21.15.116:0/3531934748 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdc040243b0 con 0x7fdc0c159710 2023-11-08T08:19:09.443 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdbf00682f0 msgr2=0x7fdbf006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdbf00682f0 0x7fdbf006a7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0b3bf0 tx=0x7fdc08009040 comp rx=0 tx=0).stop 2023-11-08T08:19:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 msgr2=0x7fdc0c157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 0x7fdc0c157c20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c06beb0 tx=0x7fdc04004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 shutdown_connections 2023-11-08T08:19:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc0c159710 0x7fdc0c157c20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdc0c070370 0x7fdc0c0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdbf00682f0 0x7fdbf006a7a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 --2- 172.21.15.116:0/3531934748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc0c0c9a50 0x7fdc0c0c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.443+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 >> 172.21.15.116:0/3531934748 conn(0x7fdc0c0cf400 msgr2=0x7fdc0c0aee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.444+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 shutdown_connections 2023-11-08T08:19:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.444+0000 7fdc135e9700 1 -- 172.21.15.116:0/3531934748 wait complete. 2023-11-08T08:19:09.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.578+0000 7fc34a9c7700 1 Processor -- start 2023-11-08T08:19:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.578+0000 7fc34a9c7700 1 -- start start 2023-11-08T08:19:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc34a9c7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc3441599b0 0x7fc34415d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc34a9c7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc34a9c7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc344159ec0 con 0x7fc3440b1650 2023-11-08T08:19:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc34a9c7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34415d8a0 con 0x7fc3441599b0 2023-11-08T08:19:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc34a9c7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34415db80 con 0x7fc3440af0a0 2023-11-08T08:19:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc3437fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc3441599b0 0x7fc34415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc348f64700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc3437fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc3441599b0 0x7fc34415d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:51158/0 (socket says 172.21.15.116:51158) 2023-11-08T08:19:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc348f64700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:50386/0 (socket says 172.21.15.116:50386) 2023-11-08T08:19:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc3437fe700 1 -- 172.21.15.116:0/1016608712 learned_addr learned my addr 172.21.15.116:0/1016608712 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.579+0000 7fc343fff700 1 --1- 172.21.15.116:0/1016608712 >> v1:172.21.15.129:6789/0 conn(0x7fc3440b1650 0x7fc3440b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:50850/0 (socket says 172.21.15.116:50850) 2023-11-08T08:19:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc3437fe700 1 -- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 msgr2=0x7fc34415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc3437fe700 1 --2- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34415a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc3437fe700 1 -- 172.21.15.116:0/1016608712 >> v1:172.21.15.129:6789/0 conn(0x7fc3440b1650 legacy=0x7fc3440b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc3437fe700 1 -- 172.21.15.116:0/1016608712 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34415de60 con 0x7fc3441599b0 2023-11-08T08:19:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc348f64700 1 --2- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc3437fe700 1 --2- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3441599b0 0x7fc34415d050 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0098f0 tx=0x7fc32c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=856262026699aef0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc342ffd700 1 -- 172.21.15.116:0/1016608712 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c011830 con 0x7fc3441599b0 2023-11-08T08:19:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc342ffd700 1 -- 172.21.15.116:0/1016608712 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c011990 con 0x7fc3441599b0 2023-11-08T08:19:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.580+0000 7fc342ffd700 1 -- 172.21.15.116:0/1016608712 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c011d20 con 0x7fc3441599b0 2023-11-08T08:19:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 -- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3441599b0 msgr2=0x7fc34415d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3441599b0 0x7fc34415d050 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0098f0 tx=0x7fc32c009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 -- 172.21.15.116:0/1016608712 shutdown_connections 2023-11-08T08:19:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/1016608712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3441599b0 0x7fc34415d050 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 -- 172.21.15.116:0/1016608712 >> 172.21.15.116:0/1016608712 conn(0x7fc34406c010 msgr2=0x7fc3440cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 -- 172.21.15.116:0/1016608712 shutdown_connections 2023-11-08T08:19:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.581+0000 7fc34a9c7700 1 -- 172.21.15.116:0/1016608712 wait complete. 2023-11-08T08:19:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.582+0000 7fc34a9c7700 1 Processor -- start 2023-11-08T08:19:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.582+0000 7fc34a9c7700 1 -- start start 2023-11-08T08:19:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc34a9c7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34414b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc34a9c7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3440b1650 0x7fc3441f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc34a9c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 0x7fc3441f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc34a9c7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34414b960 con 0x7fc3441f5870 2023-11-08T08:19:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc34a9c7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34414af90 con 0x7fc3440b1650 2023-11-08T08:19:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc34a9c7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3441f2150 con 0x7fc3440af0a0 2023-11-08T08:19:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 0x7fc3441f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 0x7fc3441f5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47922/0 (socket says 172.21.15.116:47922) 2023-11-08T08:19:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 -- 172.21.15.116:0/3337551560 learned_addr learned my addr 172.21.15.116:0/3337551560 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc343fff700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34414b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc3437fe700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3440b1650 0x7fc3441f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 -- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 msgr2=0x7fc34414b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34414b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 -- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3440b1650 msgr2=0x7fc3441f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3440b1650 0x7fc3441f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 -- 172.21.15.116:0/3337551560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c009a70 con 0x7fc3441f5870 2023-11-08T08:19:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc3437fe700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3440b1650 0x7fc3441f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc348f64700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 0x7fc3441f5ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009e60 tx=0x7fc33c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.583+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c009cd0 con 0x7fc3441f5870 2023-11-08T08:19:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.584+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33c00f040 con 0x7fc3441f5870 2023-11-08T08:19:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.584+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c009550 con 0x7fc3441f5870 2023-11-08T08:19:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.584+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c00cdc0 con 0x7fc3441f5870 2023-11-08T08:19:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.584+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34415a890 con 0x7fc3441f5870 2023-11-08T08:19:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.584+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34415d590 con 0x7fc3441f5870 2023-11-08T08:19:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.585+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc33c014480 con 0x7fc3441f5870 2023-11-08T08:19:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.585+0000 7fc340ff9700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3240682f0 0x7fc32406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.585+0000 7fc343fff700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3240682f0 0x7fc32406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.585+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc33c0864d0 con 0x7fc3441f5870 2023-11-08T08:19:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.586+0000 7fc343fff700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3240682f0 0x7fc32406a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc334000f60 tx=0x7fc334007d10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.590+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc33c052cd0 con 0x7fc3441f5870 2023-11-08T08:19:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.767+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc34415aa20 con 0x7fc3441f5870 2023-11-08T08:19:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.768+0000 7fc340ff9700 1 -- 172.21.15.116:0/3337551560 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc33c0562f0 con 0x7fc3441f5870 2023-11-08T08:19:09.770 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3240682f0 msgr2=0x7fc32406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3240682f0 0x7fc32406a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc334000f60 tx=0x7fc334007d10 comp rx=0 tx=0).stop 2023-11-08T08:19:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 msgr2=0x7fc3441f5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 0x7fc3441f5ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009e60 tx=0x7fc33c00c3a0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 shutdown_connections 2023-11-08T08:19:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3441f5870 0x7fc3441f5ce0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3440af0a0 0x7fc34414b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3240682f0 0x7fc32406a7a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 --2- 172.21.15.116:0/3337551560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3440b1650 0x7fc3441f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 >> 172.21.15.116:0/3337551560 conn(0x7fc34406c010 msgr2=0x7fc34406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 shutdown_connections 2023-11-08T08:19:09.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.770+0000 7fc34a9c7700 1 -- 172.21.15.116:0/3337551560 wait complete. 2023-11-08T08:19:09.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 Processor -- start 2023-11-08T08:19:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 -- start start 2023-11-08T08:19:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f249006ab10 0x7f249006aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24900cd5a0 0x7f2490160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24900d1ee0 con 0x7f24900b22a0 2023-11-08T08:19:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249006b420 con 0x7f24900cd5a0 2023-11-08T08:19:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f2495536700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24900cdb10 con 0x7f249006ab10 2023-11-08T08:19:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.903+0000 7f248e7fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24900cd5a0 0x7f2490160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24900cd5a0 0x7f2490160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47930/0 (socket says 172.21.15.116:47930) 2023-11-08T08:19:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248f7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f24900b22a0 0x7f2490160840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49200/0 (socket says 172.21.15.116:49200) 2023-11-08T08:19:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 -- 172.21.15.116:0/3111585759 learned_addr learned my addr 172.21.15.116:0/3111585759 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248effd700 1 --2- 172.21.15.116:0/3111585759 >> v2:172.21.15.129:3300/0 conn(0x7f249006ab10 0x7f249006aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 -- 172.21.15.116:0/3111585759 >> v2:172.21.15.129:3300/0 conn(0x7f249006ab10 msgr2=0x7f249006aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 --2- 172.21.15.116:0/3111585759 >> v2:172.21.15.129:3300/0 conn(0x7f249006ab10 0x7f249006aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 -- 172.21.15.116:0/3111585759 >> v1:172.21.15.116:6789/0 conn(0x7f24900b22a0 legacy=0x7f2490160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248dffb700 1 -- 172.21.15.116:0/3111585759 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329951353 0 0) 0x7f24900d1ee0 con 0x7f24900b22a0 2023-11-08T08:19:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 -- 172.21.15.116:0/3111585759 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900d1890 con 0x7f24900cd5a0 2023-11-08T08:19:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248e7fc700 1 --2- 172.21.15.116:0/3111585759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900cd5a0 0x7f2490160300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f24900d0cb0 tx=0x7f2480009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed004472251de431 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248dffb700 1 -- 172.21.15.116:0/3111585759 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000b870 con 0x7f24900cd5a0 2023-11-08T08:19:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248dffb700 1 -- 172.21.15.116:0/3111585759 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248000b9d0 con 0x7f24900cd5a0 2023-11-08T08:19:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.904+0000 7f248dffb700 1 -- 172.21.15.116:0/3111585759 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000bd60 con 0x7f24900cd5a0 2023-11-08T08:19:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 -- 172.21.15.116:0/3111585759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900cd5a0 msgr2=0x7f2490160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 --2- 172.21.15.116:0/3111585759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900cd5a0 0x7f2490160300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f24900d0cb0 tx=0x7f2480009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 -- 172.21.15.116:0/3111585759 shutdown_connections 2023-11-08T08:19:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 --2- 172.21.15.116:0/3111585759 >> v2:172.21.15.129:3300/0 conn(0x7f249006ab10 0x7f249006aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 --2- 172.21.15.116:0/3111585759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900cd5a0 0x7f2490160300 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 -- 172.21.15.116:0/3111585759 >> 172.21.15.116:0/3111585759 conn(0x7f249006c520 msgr2=0x7f249006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 -- 172.21.15.116:0/3111585759 shutdown_connections 2023-11-08T08:19:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.905+0000 7f2495536700 1 -- 172.21.15.116:0/3111585759 wait complete. 2023-11-08T08:19:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.906+0000 7f2495536700 1 Processor -- start 2023-11-08T08:19:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.906+0000 7f2495536700 1 -- start start 2023-11-08T08:19:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f249006ab10 0x7f24901f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 0x7f24900cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24900cd5a0 0x7f24900cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24900ca0c0 con 0x7f24900b22a0 2023-11-08T08:19:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24900ca330 con 0x7f249006ab10 2023-11-08T08:19:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24900ca610 con 0x7f24900cd5a0 2023-11-08T08:19:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 0x7f24900cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 0x7f24900cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47940/0 (socket says 172.21.15.116:47940) 2023-11-08T08:19:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 -- 172.21.15.116:0/718841025 learned_addr learned my addr 172.21.15.116:0/718841025 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248effd700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f249006ab10 0x7f24901f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248f7fe700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24900cd5a0 0x7f24900cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 -- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24900cd5a0 msgr2=0x7f24900cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24900cd5a0 0x7f24900cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 -- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f249006ab10 msgr2=0x7f24901f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f249006ab10 0x7f24901f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 -- 172.21.15.116:0/718841025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2480009550 con 0x7f24900b22a0 2023-11-08T08:19:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f248e7fc700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 0x7f24900cc330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2480009c10 tx=0x7f2480004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480004590 con 0x7f24900b22a0 2023-11-08T08:19:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24800036a0 con 0x7f24900b22a0 2023-11-08T08:19:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249015cad0 con 0x7f24900b22a0 2023-11-08T08:19:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480003a50 con 0x7f24900b22a0 2023-11-08T08:19:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.907+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24900d1b70 con 0x7f24900b22a0 2023-11-08T08:19:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.908+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24900b1ec0 con 0x7f24900b22a0 2023-11-08T08:19:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.909+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f248002a020 con 0x7f24900b22a0 2023-11-08T08:19:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.909+0000 7f24777fe700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f247c0682f0 0x7f247c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.909+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f24800872d0 con 0x7f24900b22a0 2023-11-08T08:19:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.909+0000 7f248effd700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f247c0682f0 0x7f247c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.909+0000 7f248effd700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f247c0682f0 0x7f247c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f24900c9410 tx=0x7f2484008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:09.913+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2480053b30 con 0x7f24900b22a0 2023-11-08T08:19:10.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.094+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24900743f0 con 0x7f24900b22a0 2023-11-08T08:19:10.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.095+0000 7f24777fe700 1 -- 172.21.15.116:0/718841025 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f2480057150 con 0x7f24900b22a0 2023-11-08T08:19:10.096 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:10.097 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f247c0682f0 msgr2=0x7f247c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f247c0682f0 0x7f247c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f24900c9410 tx=0x7f2484008040 comp rx=0 tx=0).stop 2023-11-08T08:19:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 msgr2=0x7f24900cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 0x7f24900cc330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2480009c10 tx=0x7f2480004750 comp rx=0 tx=0).stop 2023-11-08T08:19:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 shutdown_connections 2023-11-08T08:19:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24900cd5a0 0x7f24900cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f247c0682f0 0x7f247c06a7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f249006ab10 0x7f24901f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 --2- 172.21.15.116:0/718841025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24900b22a0 0x7f24900cc330 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 >> 172.21.15.116:0/718841025 conn(0x7f249006c520 msgr2=0x7f24900b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 shutdown_connections 2023-11-08T08:19:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.097+0000 7f2495536700 1 -- 172.21.15.116:0/718841025 wait complete. 2023-11-08T08:19:10.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:10.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.233+0000 7f18cc806700 1 Processor -- start 2023-11-08T08:19:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.233+0000 7f18cc806700 1 -- start start 2023-11-08T08:19:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.233+0000 7f18cc806700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18c8160300 0x7f18c815d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.233+0000 7f18cc806700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c815ceb0 con 0x7f18c80cec00 2023-11-08T08:19:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18cc806700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c81608c0 con 0x7f18c815d880 2023-11-08T08:19:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18cc806700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c8160ba0 con 0x7f18c8160300 2023-11-08T08:19:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c5d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18c8160300 0x7f18c815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c659c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f18c80cec00 0x7f18c8065fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49210/0 (socket says 172.21.15.116:49210) 2023-11-08T08:19:10.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c659c700 1 -- 172.21.15.116:0/877270336 learned_addr learned my addr 172.21.15.116:0/877270336 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:10.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c5d9b700 1 -- 172.21.15.116:0/877270336 >> v1:172.21.15.129:6789/0 conn(0x7f18c815d880 legacy=0x7f18c80cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:10.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c5d9b700 1 -- 172.21.15.116:0/877270336 >> v1:172.21.15.116:6789/0 conn(0x7f18c80cec00 legacy=0x7f18c8065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c5d9b700 1 -- 172.21.15.116:0/877270336 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c8160e80 con 0x7f18c8160300 2023-11-08T08:19:10.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c5d9b700 1 --2- 172.21.15.116:0/877270336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8160300 0x7f18c815d340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009840 tx=0x7f18b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3632631775f99 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c559a700 1 -- 172.21.15.116:0/877270336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8004230 con 0x7f18c8160300 2023-11-08T08:19:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c559a700 1 -- 172.21.15.116:0/877270336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b8004390 con 0x7f18c8160300 2023-11-08T08:19:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.234+0000 7f18c559a700 1 -- 172.21.15.116:0/877270336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b80036a0 con 0x7f18c8160300 2023-11-08T08:19:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 -- 172.21.15.116:0/877270336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8160300 msgr2=0x7f18c815d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 --2- 172.21.15.116:0/877270336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8160300 0x7f18c815d340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009840 tx=0x7f18b8009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 -- 172.21.15.116:0/877270336 shutdown_connections 2023-11-08T08:19:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 --2- 172.21.15.116:0/877270336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8160300 0x7f18c815d340 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 -- 172.21.15.116:0/877270336 >> 172.21.15.116:0/877270336 conn(0x7f18c80cf400 msgr2=0x7f18c80cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 -- 172.21.15.116:0/877270336 shutdown_connections 2023-11-08T08:19:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.235+0000 7f18cc806700 1 -- 172.21.15.116:0/877270336 wait complete. 2023-11-08T08:19:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.236+0000 7f18cc806700 1 Processor -- start 2023-11-08T08:19:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.236+0000 7f18cc806700 1 -- start start 2023-11-08T08:19:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18cc806700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c80cec00 0x7f18c81e7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18cc806700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 0x7f18c81506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18cc806700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18c8160300 0x7f18c80ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18cc806700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c81e8170 con 0x7f18c815d880 2023-11-08T08:19:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18cc806700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c81e77a0 con 0x7f18c80cec00 2023-11-08T08:19:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18cc806700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c8150c30 con 0x7f18c8160300 2023-11-08T08:19:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 0x7f18c81506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 0x7f18c81506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47946/0 (socket says 172.21.15.116:47946) 2023-11-08T08:19:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 -- 172.21.15.116:0/2902953631 learned_addr learned my addr 172.21.15.116:0/2902953631 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 -- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18c8160300 msgr2=0x7f18c80ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18c8160300 0x7f18c80ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 -- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c80cec00 msgr2=0x7f18c81e7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c659c700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c80cec00 0x7f18c81e7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c80cec00 0x7f18c81e7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 -- 172.21.15.116:0/2902953631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b80094a0 con 0x7f18c815d880 2023-11-08T08:19:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.237+0000 7f18c5d9b700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 0x7f18c81506f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f18c806beb0 tx=0x7f18b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.238+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8011630 con 0x7f18c815d880 2023-11-08T08:19:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.238+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b801e070 con 0x7f18c815d880 2023-11-08T08:19:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.238+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c80cefd0 con 0x7f18c815d880 2023-11-08T08:19:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.238+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b80119b0 con 0x7f18c815d880 2023-11-08T08:19:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.238+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c8161160 con 0x7f18c815d880 2023-11-08T08:19:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.239+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c80cf160 con 0x7f18c815d880 2023-11-08T08:19:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.239+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18b8017020 con 0x7f18c815d880 2023-11-08T08:19:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.239+0000 7f18b6ffd700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18a80682f0 0x7f18a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.239+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f18b8087020 con 0x7f18c815d880 2023-11-08T08:19:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.239+0000 7f18c659c700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18a80682f0 0x7f18a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.240+0000 7f18c659c700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18a80682f0 0x7f18a806a7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000f60 tx=0x7f18bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.243+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18b8053880 con 0x7f18c815d880 2023-11-08T08:19:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.427+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f18c80743f0 con 0x7f18c815d880 2023-11-08T08:19:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.427+0000 7f18b6ffd700 1 -- 172.21.15.116:0/2902953631 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f18b80200a0 con 0x7f18c815d880 2023-11-08T08:19:10.428 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18a80682f0 msgr2=0x7f18a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18a80682f0 0x7f18a806a7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000f60 tx=0x7f18bc009040 comp rx=0 tx=0).stop 2023-11-08T08:19:10.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 msgr2=0x7f18c81506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 0x7f18c81506f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f18c806beb0 tx=0x7f18b8004750 comp rx=0 tx=0).stop 2023-11-08T08:19:10.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 shutdown_connections 2023-11-08T08:19:10.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18c8160300 0x7f18c80ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18a80682f0 0x7f18a806a7a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c80cec00 0x7f18c81e7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 --2- 172.21.15.116:0/2902953631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c815d880 0x7f18c81506f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 >> 172.21.15.116:0/2902953631 conn(0x7f18c80cf400 msgr2=0x7f18c80af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 shutdown_connections 2023-11-08T08:19:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.429+0000 7f18cc806700 1 -- 172.21.15.116:0/2902953631 wait complete. 2023-11-08T08:19:10.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:10.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.564+0000 7f9a7bc30700 1 Processor -- start 2023-11-08T08:19:10.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.564+0000 7f9a7bc30700 1 -- start start 2023-11-08T08:19:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7bc30700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a7406a980 0x7f9a740ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7bc30700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a74071890 0x7f9a74071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7bc30700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7415cc10 con 0x7f9a740cf710 2023-11-08T08:19:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7bc30700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7406c9b0 con 0x7f9a74071890 2023-11-08T08:19:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7bc30700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7406cc90 con 0x7f9a7406a980 2023-11-08T08:19:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a74071890 0x7f9a74071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a74071890 0x7f9a74071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47958/0 (socket says 172.21.15.116:47958) 2023-11-08T08:19:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 -- 172.21.15.116:0/550140210 learned_addr learned my addr 172.21.15.116:0/550140210 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 -- 172.21.15.116:0/550140210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a7406a980 msgr2=0x7f9a740ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 --2- 172.21.15.116:0/550140210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a7406a980 0x7f9a740ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 -- 172.21.15.116:0/550140210 >> v1:172.21.15.129:6789/0 conn(0x7f9a740cf710 legacy=0x7f9a7406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.565+0000 7f9a7a1cd700 1 -- 172.21.15.116:0/550140210 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7406cf70 con 0x7f9a74071890 2023-11-08T08:19:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7a1cd700 1 --2- 172.21.15.116:0/550140210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a74071890 0x7f9a74071220 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009e50 tx=0x7f9a6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca894b7d4d54295f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a789ca700 1 -- 172.21.15.116:0/550140210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00e070 con 0x7f9a74071890 2023-11-08T08:19:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a789ca700 1 -- 172.21.15.116:0/550140210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c003a20 con 0x7f9a74071890 2023-11-08T08:19:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a789ca700 1 -- 172.21.15.116:0/550140210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c003dd0 con 0x7f9a74071890 2023-11-08T08:19:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 -- 172.21.15.116:0/550140210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a74071890 msgr2=0x7f9a74071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/550140210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a74071890 0x7f9a74071220 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009e50 tx=0x7f9a6c00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 -- 172.21.15.116:0/550140210 shutdown_connections 2023-11-08T08:19:10.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/550140210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a74071890 0x7f9a74071220 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/550140210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a7406a980 0x7f9a740ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 -- 172.21.15.116:0/550140210 >> 172.21.15.116:0/550140210 conn(0x7f9a7406c010 msgr2=0x7f9a740cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:10.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 -- 172.21.15.116:0/550140210 shutdown_connections 2023-11-08T08:19:10.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.566+0000 7f9a7bc30700 1 -- 172.21.15.116:0/550140210 wait complete. 2023-11-08T08:19:10.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 Processor -- start 2023-11-08T08:19:10.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 -- start start 2023-11-08T08:19:10.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 0x7f9a740c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a74071890 0x7f9a741f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740cf710 0x7f9a740b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740c8980 con 0x7f9a7406a980 2023-11-08T08:19:10.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a741f61c0 con 0x7f9a740cf710 2023-11-08T08:19:10.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a7bc30700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7415de50 con 0x7f9a74071890 2023-11-08T08:19:10.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a799cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 0x7f9a740c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a799cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 0x7f9a740c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47962/0 (socket says 172.21.15.116:47962) 2023-11-08T08:19:10.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.568+0000 7f9a799cc700 1 -- 172.21.15.116:0/3927844967 learned_addr learned my addr 172.21.15.116:0/3927844967 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:10.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a7a1cd700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740cf710 0x7f9a740b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a791cb700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a74071890 0x7f9a741f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a799cc700 1 -- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a74071890 msgr2=0x7f9a741f5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a799cc700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a74071890 0x7f9a741f5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a799cc700 1 -- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740cf710 msgr2=0x7f9a740b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a799cc700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740cf710 0x7f9a740b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a799cc700 1 -- 172.21.15.116:0/3927844967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c009b30 con 0x7f9a7406a980 2023-11-08T08:19:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a799cc700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 0x7f9a740c8d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400b470 tx=0x7f9a6400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64011830 con 0x7f9a7406a980 2023-11-08T08:19:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64011990 con 0x7f9a7406a980 2023-11-08T08:19:10.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7406d530 con 0x7f9a7406a980 2023-11-08T08:19:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64011d40 con 0x7f9a7406a980 2023-11-08T08:19:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.569+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7406d250 con 0x7f9a7406a980 2023-11-08T08:19:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.570+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74160480 con 0x7f9a7406a980 2023-11-08T08:19:10.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.570+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a64021070 con 0x7f9a7406a980 2023-11-08T08:19:10.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.571+0000 7f9a6a7fc700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9a580682f0 0x7f9a5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.571+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9a64085b90 con 0x7f9a7406a980 2023-11-08T08:19:10.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.571+0000 7f9a791cb700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9a580682f0 0x7f9a5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.571+0000 7f9a791cb700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9a580682f0 0x7f9a5806a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006fd0 tx=0x7f9a70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.575+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a64052390 con 0x7f9a7406a980 2023-11-08T08:19:10.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.752+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a74074440 con 0x7f9a7406a980 2023-11-08T08:19:10.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.752+0000 7f9a6a7fc700 1 -- 172.21.15.116:0/3927844967 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a64015020 con 0x7f9a7406a980 2023-11-08T08:19:10.754 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9a580682f0 msgr2=0x7f9a5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9a580682f0 0x7f9a5806a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006fd0 tx=0x7f9a70008040 comp rx=0 tx=0).stop 2023-11-08T08:19:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 msgr2=0x7f9a740c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 0x7f9a740c8d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400b470 tx=0x7f9a6400b830 comp rx=0 tx=0).stop 2023-11-08T08:19:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 shutdown_connections 2023-11-08T08:19:10.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9a580682f0 0x7f9a5806a7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740cf710 0x7f9a740b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7406a980 0x7f9a740c8d90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 --2- 172.21.15.116:0/3927844967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9a74071890 0x7f9a741f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 >> 172.21.15.116:0/3927844967 conn(0x7f9a7406c010 msgr2=0x7f9a740b5150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:10.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 shutdown_connections 2023-11-08T08:19:10.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.754+0000 7f9a7bc30700 1 -- 172.21.15.116:0/3927844967 wait complete. 2023-11-08T08:19:10.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:10.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.889+0000 7f29e685b700 1 Processor -- start 2023-11-08T08:19:10.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.889+0000 7f29e685b700 1 -- start start 2023-11-08T08:19:10.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29e685b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f29e0160cb0 0x7f29e006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29e685b700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e00d2d30 con 0x7f29e015c8c0 2023-11-08T08:19:10.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29e685b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e00d0b40 con 0x7f29e006b050 2023-11-08T08:19:10.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29e685b700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e006b490 con 0x7f29e0160cb0 2023-11-08T08:19:10.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29df7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f29e0160cb0 0x7f29e006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29dffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f29e015c8c0 0x7f29e00b5dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49220/0 (socket says 172.21.15.116:49220) 2023-11-08T08:19:10.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29dffff700 1 -- 172.21.15.116:0/3964372185 learned_addr learned my addr 172.21.15.116:0/3964372185 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:10.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29df7fe700 1 -- 172.21.15.116:0/3964372185 >> v1:172.21.15.129:6789/0 conn(0x7f29e006b050 legacy=0x7f29e00d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:10.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29df7fe700 1 -- 172.21.15.116:0/3964372185 >> v1:172.21.15.116:6789/0 conn(0x7f29e015c8c0 legacy=0x7f29e00b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29deffd700 1 -- 172.21.15.116:0/3964372185 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016351642 0 0) 0x7f29d4001110 con 0x7f29e015c8c0 2023-11-08T08:19:10.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29df7fe700 1 -- 172.21.15.116:0/3964372185 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00d2a50 con 0x7f29e0160cb0 2023-11-08T08:19:10.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29df7fe700 1 --2- 172.21.15.116:0/3964372185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0160cb0 0x7f29e006ab10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009840 tx=0x7f29d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f30c7e7fa70665 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.890+0000 7f29deffd700 1 -- 172.21.15.116:0/3964372185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0011830 con 0x7f29e0160cb0 2023-11-08T08:19:10.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29deffd700 1 -- 172.21.15.116:0/3964372185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d0011990 con 0x7f29e0160cb0 2023-11-08T08:19:10.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29deffd700 1 -- 172.21.15.116:0/3964372185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0011d20 con 0x7f29e0160cb0 2023-11-08T08:19:10.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 -- 172.21.15.116:0/3964372185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0160cb0 msgr2=0x7f29e006ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 --2- 172.21.15.116:0/3964372185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0160cb0 0x7f29e006ab10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009840 tx=0x7f29d0009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:10.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 -- 172.21.15.116:0/3964372185 shutdown_connections 2023-11-08T08:19:10.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 --2- 172.21.15.116:0/3964372185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0160cb0 0x7f29e006ab10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 -- 172.21.15.116:0/3964372185 >> 172.21.15.116:0/3964372185 conn(0x7f29e006c520 msgr2=0x7f29e006c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:10.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 -- 172.21.15.116:0/3964372185 shutdown_connections 2023-11-08T08:19:10.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.891+0000 7f29e685b700 1 -- 172.21.15.116:0/3964372185 wait complete. 2023-11-08T08:19:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.892+0000 7f29e685b700 1 Processor -- start 2023-11-08T08:19:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.892+0000 7f29e685b700 1 -- start start 2023-11-08T08:19:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29e685b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 0x7f29e015f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29e685b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29e015c8c0 0x7f29e00c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29e685b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29e0160cb0 0x7f29e01507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29e685b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e014f6d0 con 0x7f29e006b050 2023-11-08T08:19:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29e685b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00c9780 con 0x7f29e0160cb0 2023-11-08T08:19:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29e685b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e014eac0 con 0x7f29e015c8c0 2023-11-08T08:19:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29dffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 0x7f29e015f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29dffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 0x7f29e015f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47984/0 (socket says 172.21.15.116:47984) 2023-11-08T08:19:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.893+0000 7f29dffff700 1 -- 172.21.15.116:0/3895262694 learned_addr learned my addr 172.21.15.116:0/3895262694 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dffff700 1 -- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29e015c8c0 msgr2=0x7f29e00c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29df7fe700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29e015c8c0 0x7f29e00c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29e4df8700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29e0160cb0 0x7f29e01507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dffff700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29e015c8c0 0x7f29e00c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dffff700 1 -- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29e0160cb0 msgr2=0x7f29e01507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dffff700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29e0160cb0 0x7f29e01507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dffff700 1 -- 172.21.15.116:0/3895262694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d00094a0 con 0x7f29e006b050 2023-11-08T08:19:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dffff700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 0x7f29e015f4f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f29d400c550 tx=0x7f29d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4004c80 con 0x7f29e006b050 2023-11-08T08:19:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4004de0 con 0x7f29e006b050 2023-11-08T08:19:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e00cda40 con 0x7f29e006b050 2023-11-08T08:19:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4007310 con 0x7f29e006b050 2023-11-08T08:19:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.894+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e00cd760 con 0x7f29e006b050 2023-11-08T08:19:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.895+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e0161120 con 0x7f29e006b050 2023-11-08T08:19:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.895+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f29d401a020 con 0x7f29e006b050 2023-11-08T08:19:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.896+0000 7f29dcff9700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29cc0682f0 0x7f29cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.896+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f29d4085790 con 0x7f29e006b050 2023-11-08T08:19:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.896+0000 7f29df7fe700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29cc0682f0 0x7f29cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.896+0000 7f29df7fe700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29cc0682f0 0x7f29cc06a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f29e00b8ad0 tx=0x7f29d001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:10.900+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29d4051f00 con 0x7f29e006b050 2023-11-08T08:19:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.078+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29e0067d70 con 0x7f29e006b050 2023-11-08T08:19:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.079+0000 7f29dcff9700 1 -- 172.21.15.116:0/3895262694 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29d4055520 con 0x7f29e006b050 2023-11-08T08:19:11.080 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29cc0682f0 msgr2=0x7f29cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29cc0682f0 0x7f29cc06a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f29e00b8ad0 tx=0x7f29d001c040 comp rx=0 tx=0).stop 2023-11-08T08:19:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 msgr2=0x7f29e015f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 0x7f29e015f4f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f29d400c550 tx=0x7f29d400c910 comp rx=0 tx=0).stop 2023-11-08T08:19:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 shutdown_connections 2023-11-08T08:19:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29cc0682f0 0x7f29cc06a7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29e0160cb0 0x7f29e01507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e006b050 0x7f29e015f4f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 --2- 172.21.15.116:0/3895262694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29e015c8c0 0x7f29e00c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 >> 172.21.15.116:0/3895262694 conn(0x7f29e006c520 msgr2=0x7f29e006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 shutdown_connections 2023-11-08T08:19:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.081+0000 7f29e685b700 1 -- 172.21.15.116:0/3895262694 wait complete. 2023-11-08T08:19:11.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.215+0000 7ff8416f1700 1 Processor -- start 2023-11-08T08:19:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.215+0000 7ff8416f1700 1 -- start start 2023-11-08T08:19:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff8416f1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c15a590 0x7ff83c06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff8416f1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 0x7ff83c06a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff8416f1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0cd480 con 0x7ff83c0b50d0 2023-11-08T08:19:11.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff8416f1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0cd760 con 0x7ff83c06cb50 2023-11-08T08:19:11.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff8416f1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0cda40 con 0x7ff83c15a590 2023-11-08T08:19:11.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff83b7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 0x7ff83c06a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff83affd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c15a590 0x7ff83c06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff83b7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 0x7ff83c06a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:51218/0 (socket says 172.21.15.116:51218) 2023-11-08T08:19:11.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.216+0000 7ff83b7fe700 1 -- 172.21.15.116:0/1918660139 learned_addr learned my addr 172.21.15.116:0/1918660139 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:11.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff83affd700 1 -- 172.21.15.116:0/1918660139 >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 msgr2=0x7ff83c06a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:11.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff83affd700 1 --2- 172.21.15.116:0/1918660139 >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 0x7ff83c06a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff83affd700 1 -- 172.21.15.116:0/1918660139 >> v1:172.21.15.129:6789/0 conn(0x7ff83c0b50d0 legacy=0x7ff83c06c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff83affd700 1 -- 172.21.15.116:0/1918660139 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0cdd20 con 0x7ff83c15a590 2023-11-08T08:19:11.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff83b7fe700 1 --2- 172.21.15.116:0/1918660139 >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 0x7ff83c06a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:11.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff83affd700 1 --2- 172.21.15.116:0/1918660139 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c15a590 0x7ff83c06b5b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff830009840 tx=0x7ff830009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88b380b3dec1a7cc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:11.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff839ffb700 1 -- 172.21.15.116:0/1918660139 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff830011830 con 0x7ff83c15a590 2023-11-08T08:19:11.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff839ffb700 1 -- 172.21.15.116:0/1918660139 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff830011990 con 0x7ff83c15a590 2023-11-08T08:19:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.217+0000 7ff839ffb700 1 -- 172.21.15.116:0/1918660139 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff830011d20 con 0x7ff83c15a590 2023-11-08T08:19:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 -- 172.21.15.116:0/1918660139 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c15a590 msgr2=0x7ff83c06b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 --2- 172.21.15.116:0/1918660139 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c15a590 0x7ff83c06b5b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff830009840 tx=0x7ff830009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 -- 172.21.15.116:0/1918660139 shutdown_connections 2023-11-08T08:19:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 --2- 172.21.15.116:0/1918660139 >> v2:172.21.15.129:3300/0 conn(0x7ff83c06cb50 0x7ff83c06a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 --2- 172.21.15.116:0/1918660139 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c15a590 0x7ff83c06b5b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 -- 172.21.15.116:0/1918660139 >> 172.21.15.116:0/1918660139 conn(0x7ff83c0cf400 msgr2=0x7ff83c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 -- 172.21.15.116:0/1918660139 shutdown_connections 2023-11-08T08:19:11.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.218+0000 7ff8416f1700 1 -- 172.21.15.116:0/1918660139 wait complete. 2023-11-08T08:19:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.219+0000 7ff8416f1700 1 Processor -- start 2023-11-08T08:19:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 -- start start 2023-11-08T08:19:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 0x7ff83c150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c0b50d0 0x7ff83c0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff83c15a590 0x7ff83c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0c4c90 con 0x7ff83c06cb50 2023-11-08T08:19:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0c4f70 con 0x7ff83c15a590 2023-11-08T08:19:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff8416f1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0c5250 con 0x7ff83c0b50d0 2023-11-08T08:19:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.220+0000 7ff83affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 0x7ff83c150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 0x7ff83c150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47988/0 (socket says 172.21.15.116:47988) 2023-11-08T08:19:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 -- 172.21.15.116:0/4200067884 learned_addr learned my addr 172.21.15.116:0/4200067884 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 -- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c0b50d0 msgr2=0x7ff83c0c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83a7fc700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c0b50d0 0x7ff83c0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c0b50d0 0x7ff83c0c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83b7fe700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff83c15a590 0x7ff83c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 -- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff83c15a590 msgr2=0x7ff83c15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff83c15a590 0x7ff83c15b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 -- 172.21.15.116:0/4200067884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834009b30 con 0x7ff83c06cb50 2023-11-08T08:19:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff83affd700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 0x7ff83c150c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff830009fd0 tx=0x7ff830004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8300036a0 con 0x7ff83c06cb50 2023-11-08T08:19:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83001e070 con 0x7ff83c06cb50 2023-11-08T08:19:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8300094a0 con 0x7ff83c06cb50 2023-11-08T08:19:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff830003a60 con 0x7ff83c06cb50 2023-11-08T08:19:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.221+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c06baf0 con 0x7ff83c06cb50 2023-11-08T08:19:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.222+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c0ce000 con 0x7ff83c06cb50 2023-11-08T08:19:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.222+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff83000f050 con 0x7ff83c06cb50 2023-11-08T08:19:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.223+0000 7ff8237fe700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff8280682f0 0x7ff82806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.223+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff830086d20 con 0x7ff83c06cb50 2023-11-08T08:19:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.223+0000 7ff83a7fc700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff8280682f0 0x7ff82806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.223+0000 7ff83a7fc700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff8280682f0 0x7ff82806a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff82c007fd0 tx=0x7ff82c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.227+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff830053490 con 0x7ff83c06cb50 2023-11-08T08:19:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.406+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff83c15a960 con 0x7ff83c06cb50 2023-11-08T08:19:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.407+0000 7ff8237fe700 1 -- 172.21.15.116:0/4200067884 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff8300150a0 con 0x7ff83c06cb50 2023-11-08T08:19:11.408 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff8280682f0 msgr2=0x7ff82806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff8280682f0 0x7ff82806a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff82c007fd0 tx=0x7ff82c009040 comp rx=0 tx=0).stop 2023-11-08T08:19:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 msgr2=0x7ff83c150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 0x7ff83c150c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff830009fd0 tx=0x7ff830004750 comp rx=0 tx=0).stop 2023-11-08T08:19:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 shutdown_connections 2023-11-08T08:19:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff8280682f0 0x7ff82806a7a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff83c15a590 0x7ff83c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff83c06cb50 0x7ff83c150c40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 --2- 172.21.15.116:0/4200067884 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff83c0b50d0 0x7ff83c0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 >> 172.21.15.116:0/4200067884 conn(0x7ff83c0cf400 msgr2=0x7ff83c158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 shutdown_connections 2023-11-08T08:19:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.409+0000 7ff8416f1700 1 -- 172.21.15.116:0/4200067884 wait complete. 2023-11-08T08:19:11.430 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:19:11.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2023-11-08T08:19:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.551+0000 7f55f0dbe700 1 Processor -- start 2023-11-08T08:19:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.551+0000 7f55f0dbe700 1 -- start start 2023-11-08T08:19:11.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55f0dbe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f55ec0ce040 0x7f55ec15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55f0dbe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f55ec142db0 0x7f55ec13ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55f0dbe700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ec13fc90 con 0x7f55ec0b1770 2023-11-08T08:19:11.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55f0dbe700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ec13f6d0 con 0x7f55ec0ce040 2023-11-08T08:19:11.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55f0dbe700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ec13f9b0 con 0x7f55ec142db0 2023-11-08T08:19:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f55ec0ce040 0x7f55ec15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ead9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f55ec0b1770 0x7f55ec0b1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49228/0 (socket says 172.21.15.116:49228) 2023-11-08T08:19:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f55ec0ce040 0x7f55ec15d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47994/0 (socket says 172.21.15.116:47994) 2023-11-08T08:19:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 -- 172.21.15.116:0/3274602793 learned_addr learned my addr 172.21.15.116:0/3274602793 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55e9d9b700 1 --2- 172.21.15.116:0/3274602793 >> v2:172.21.15.129:3300/0 conn(0x7f55ec142db0 0x7f55ec13ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 -- 172.21.15.116:0/3274602793 >> v2:172.21.15.129:3300/0 conn(0x7f55ec142db0 msgr2=0x7f55ec13ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 --2- 172.21.15.116:0/3274602793 >> v2:172.21.15.129:3300/0 conn(0x7f55ec142db0 0x7f55ec13ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 -- 172.21.15.116:0/3274602793 >> v1:172.21.15.116:6789/0 conn(0x7f55ec0b1770 legacy=0x7f55ec0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 -- 172.21.15.116:0/3274602793 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec15d100 con 0x7f55ec0ce040 2023-11-08T08:19:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55e959a700 1 -- 172.21.15.116:0/3274602793 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982824613 0 0) 0x7f55ec13fc90 con 0x7f55ec0b1770 2023-11-08T08:19:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55e9d9b700 1 --2- 172.21.15.116:0/3274602793 >> v2:172.21.15.129:3300/0 conn(0x7f55ec142db0 0x7f55ec13ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.552+0000 7f55ea59c700 1 --2- 172.21.15.116:0/3274602793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0ce040 0x7f55ec15d900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0cfc20 tx=0x7f55e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fe08c50dee1174 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55e959a700 1 -- 172.21.15.116:0/3274602793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e000b870 con 0x7f55ec0ce040 2023-11-08T08:19:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55e959a700 1 -- 172.21.15.116:0/3274602793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e000b9d0 con 0x7f55ec0ce040 2023-11-08T08:19:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55e959a700 1 -- 172.21.15.116:0/3274602793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e000bd60 con 0x7f55ec0ce040 2023-11-08T08:19:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 -- 172.21.15.116:0/3274602793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0ce040 msgr2=0x7f55ec15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/3274602793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0ce040 0x7f55ec15d900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0cfc20 tx=0x7f55e0009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 -- 172.21.15.116:0/3274602793 shutdown_connections 2023-11-08T08:19:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/3274602793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0ce040 0x7f55ec15d900 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/3274602793 >> v2:172.21.15.129:3300/0 conn(0x7f55ec142db0 0x7f55ec13ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 -- 172.21.15.116:0/3274602793 >> 172.21.15.116:0/3274602793 conn(0x7f55ec06c010 msgr2=0x7f55ec0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 -- 172.21.15.116:0/3274602793 shutdown_connections 2023-11-08T08:19:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.553+0000 7f55f0dbe700 1 -- 172.21.15.116:0/3274602793 wait complete. 2023-11-08T08:19:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.554+0000 7f55f0dbe700 1 Processor -- start 2023-11-08T08:19:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 -- start start 2023-11-08T08:19:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 0x7f55ec13ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55ec0ce040 0x7f55ec1f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec142db0 0x7f55ec0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0c3a90 con 0x7f55ec0b1770 2023-11-08T08:19:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0c59b0 con 0x7f55ec142db0 2023-11-08T08:19:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55f0dbe700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0c5c90 con 0x7f55ec0ce040 2023-11-08T08:19:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55ea59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 0x7f55ec13ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55ea59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 0x7f55ec13ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48010/0 (socket says 172.21.15.116:48010) 2023-11-08T08:19:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.555+0000 7f55ea59c700 1 -- 172.21.15.116:0/426715171 learned_addr learned my addr 172.21.15.116:0/426715171 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55ea59c700 1 -- 172.21.15.116:0/426715171 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55ec0ce040 msgr2=0x7f55ec1f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55ea59c700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55ec0ce040 0x7f55ec1f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55ea59c700 1 -- 172.21.15.116:0/426715171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec142db0 msgr2=0x7f55ec0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55ea59c700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec142db0 0x7f55ec0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55ea59c700 1 -- 172.21.15.116:0/426715171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0009550 con 0x7f55ec0b1770 2023-11-08T08:19:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55ea59c700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 0x7f55ec13ffe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009c10 tx=0x7f55e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0004310 con 0x7f55ec0b1770 2023-11-08T08:19:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e001f070 con 0x7f55ec0b1770 2023-11-08T08:19:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec0ce410 con 0x7f55ec0b1770 2023-11-08T08:19:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e00036a0 con 0x7f55ec0b1770 2023-11-08T08:19:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.556+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec15d3e0 con 0x7f55ec0b1770 2023-11-08T08:19:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.557+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec0ce5a0 con 0x7f55ec0b1770 2023-11-08T08:19:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.557+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f55e002a020 con 0x7f55ec0b1770 2023-11-08T08:19:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.557+0000 7f55daffd700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55cc0682f0 0x7f55cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.558+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f55e0087310 con 0x7f55ec0b1770 2023-11-08T08:19:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.558+0000 7f55e9d9b700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55cc0682f0 0x7f55cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.558+0000 7f55e9d9b700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55cc0682f0 0x7f55cc06a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0cbd90 tx=0x7f55dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.562+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55e0053b70 con 0x7f55ec0b1770 2023-11-08T08:19:11.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:11.815+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f55ec0743f0 con 0x7f55ec0b1770 2023-11-08T08:19:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.479+0000 7f55daffd700 1 -- 172.21.15.116:0/426715171 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v45) v1 ==== 145+0+0 (secure 0 0 0) 0x7f55e0057190 con 0x7f55ec0b1770 2023-11-08T08:19:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55cc0682f0 msgr2=0x7f55cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55cc0682f0 0x7f55cc06a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0cbd90 tx=0x7f55dc008040 comp rx=0 tx=0).stop 2023-11-08T08:19:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 msgr2=0x7f55ec13ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 0x7f55ec13ffe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009c10 tx=0x7f55e0004750 comp rx=0 tx=0).stop 2023-11-08T08:19:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 shutdown_connections 2023-11-08T08:19:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55cc0682f0 0x7f55cc06a7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec142db0 0x7f55ec0cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55ec0b1770 0x7f55ec13ffe0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 --2- 172.21.15.116:0/426715171 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55ec0ce040 0x7f55ec1f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 >> 172.21.15.116:0/426715171 conn(0x7f55ec06c010 msgr2=0x7f55ec0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 shutdown_connections 2023-11-08T08:19:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.481+0000 7f55f0dbe700 1 -- 172.21.15.116:0/426715171 wait complete. 2023-11-08T08:19:12.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-08T08:19:12.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.618+0000 7f65cab11700 1 Processor -- start 2023-11-08T08:19:12.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.619+0000 7f65cab11700 1 -- start start 2023-11-08T08:19:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.619+0000 7f65cab11700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65c40b1650 0x7f65c40b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.619+0000 7f65cab11700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 0x7f65c415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.619+0000 7f65cab11700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c4159ec0 con 0x7f65c41599b0 2023-11-08T08:19:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.619+0000 7f65cab11700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c40b1f60 con 0x7f65c40b1650 2023-11-08T08:19:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.619+0000 7f65cab11700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c415d8a0 con 0x7f65c40af0a0 2023-11-08T08:19:12.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c88ad700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65c40b1650 0x7f65c40b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:12.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 0x7f65c415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:12.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 0x7f65c415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:52852/0 (socket says 172.21.15.116:52852) 2023-11-08T08:19:12.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 -- 172.21.15.116:0/870541669 learned_addr learned my addr 172.21.15.116:0/870541669 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:12.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 -- 172.21.15.116:0/870541669 >> v2:172.21.15.129:3300/0 conn(0x7f65c40b1650 msgr2=0x7f65c40b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:12.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 --2- 172.21.15.116:0/870541669 >> v2:172.21.15.129:3300/0 conn(0x7f65c40b1650 0x7f65c40b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 -- 172.21.15.116:0/870541669 >> v1:172.21.15.129:6789/0 conn(0x7f65c41599b0 legacy=0x7f65c415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 -- 172.21.15.116:0/870541669 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c415db80 con 0x7f65c40af0a0 2023-11-08T08:19:12.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c88ad700 1 --2- 172.21.15.116:0/870541669 >> v2:172.21.15.129:3300/0 conn(0x7f65c40b1650 0x7f65c40b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:12.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c90ae700 1 --2- 172.21.15.116:0/870541669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 0x7f65c415a350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009e50 tx=0x7f65bc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e76ab2127dfab05c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:12.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c37fe700 1 -- 172.21.15.116:0/870541669 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc00e070 con 0x7f65c40af0a0 2023-11-08T08:19:12.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c37fe700 1 -- 172.21.15.116:0/870541669 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc00b910 con 0x7f65c40af0a0 2023-11-08T08:19:12.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.620+0000 7f65c37fe700 1 -- 172.21.15.116:0/870541669 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc00bcc0 con 0x7f65c40af0a0 2023-11-08T08:19:12.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 -- 172.21.15.116:0/870541669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 msgr2=0x7f65c415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 --2- 172.21.15.116:0/870541669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 0x7f65c415a350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009e50 tx=0x7f65bc00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:12.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 -- 172.21.15.116:0/870541669 shutdown_connections 2023-11-08T08:19:12.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 --2- 172.21.15.116:0/870541669 >> v2:172.21.15.129:3300/0 conn(0x7f65c40b1650 0x7f65c40b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 --2- 172.21.15.116:0/870541669 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40af0a0 0x7f65c415a350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 -- 172.21.15.116:0/870541669 >> 172.21.15.116:0/870541669 conn(0x7f65c406c010 msgr2=0x7f65c40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:12.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 -- 172.21.15.116:0/870541669 shutdown_connections 2023-11-08T08:19:12.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.621+0000 7f65cab11700 1 -- 172.21.15.116:0/870541669 wait complete. 2023-11-08T08:19:12.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.622+0000 7f65cab11700 1 Processor -- start 2023-11-08T08:19:12.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.622+0000 7f65cab11700 1 -- start start 2023-11-08T08:19:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65cab11700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65c40af0a0 0x7f65c414fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65cab11700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40b1650 0x7f65c4150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65cab11700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 0x7f65c40c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65cab11700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c4150f60 con 0x7f65c41f3bf0 2023-11-08T08:19:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65cab11700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c41f4080 con 0x7f65c40af0a0 2023-11-08T08:19:12.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65cab11700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c41f4360 con 0x7f65c40b1650 2023-11-08T08:19:12.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65c90ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 0x7f65c40c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:12.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65c90ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 0x7f65c40c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35000/0 (socket says 172.21.15.116:35000) 2023-11-08T08:19:12.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65c90ae700 1 -- 172.21.15.116:0/3585822249 learned_addr learned my addr 172.21.15.116:0/3585822249 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:12.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65c3fff700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40b1650 0x7f65c4150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.623+0000 7f65c88ad700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65c40af0a0 0x7f65c414fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c90ae700 1 -- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40b1650 msgr2=0x7f65c4150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c90ae700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40b1650 0x7f65c4150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c90ae700 1 -- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65c40af0a0 msgr2=0x7f65c414fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c90ae700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65c40af0a0 0x7f65c414fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c90ae700 1 -- 172.21.15.116:0/3585822249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b80094a0 con 0x7f65c41f3bf0 2023-11-08T08:19:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c90ae700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 0x7f65c40c63a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f65c415d9e0 tx=0x7f65bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc004800 con 0x7f65c41f3bf0 2023-11-08T08:19:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc004960 con 0x7f65c41f3bf0 2023-11-08T08:19:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc009b30 con 0x7f65c41f3bf0 2023-11-08T08:19:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc01b670 con 0x7f65c41f3bf0 2023-11-08T08:19:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.624+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c415a890 con 0x7f65c41f3bf0 2023-11-08T08:19:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.625+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c415de60 con 0x7f65c41f3bf0 2023-11-08T08:19:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.625+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65bc02b070 con 0x7f65c41f3bf0 2023-11-08T08:19:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.626+0000 7f65c17fa700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65a80682f0 0x7f65a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.626+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f65bc087320 con 0x7f65c41f3bf0 2023-11-08T08:19:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.626+0000 7f65c88ad700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65a80682f0 0x7f65a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.626+0000 7f65c88ad700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65a80682f0 0x7f65a806a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009cd0 tx=0x7f65b8009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.630+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f65bc053b80 con 0x7f65c41f3bf0 2023-11-08T08:19:12.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.811+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f65c415aa20 con 0x7f65c41f3bf0 2023-11-08T08:19:12.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.815+0000 7f65c17fa700 1 -- 172.21.15.116:0/3585822249 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f65bc0571a0 con 0x7f65c41f3bf0 2023-11-08T08:19:12.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65a80682f0 msgr2=0x7f65a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65a80682f0 0x7f65a806a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009cd0 tx=0x7f65b8009210 comp rx=0 tx=0).stop 2023-11-08T08:19:12.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 msgr2=0x7f65c40c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:12.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 0x7f65c40c63a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f65c415d9e0 tx=0x7f65bc004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 shutdown_connections 2023-11-08T08:19:12.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c41f3bf0 0x7f65c40c63a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65a80682f0 0x7f65a806a7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65c40af0a0 0x7f65c414fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 --2- 172.21.15.116:0/3585822249 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f65c40b1650 0x7f65c4150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:12.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 >> 172.21.15.116:0/3585822249 conn(0x7f65c406c010 msgr2=0x7f65c4158fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:12.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 shutdown_connections 2023-11-08T08:19:12.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:12.817+0000 7f65cab11700 1 -- 172.21.15.116:0/3585822249 wait complete. 2023-11-08T08:19:12.830 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-08T08:19:12.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:19:12.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:19:12.919 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-08T08:19:12.919 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:12.920 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:12.938 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:12.939 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:12.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:13.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.157+0000 7f0a52d72700 1 Processor -- start 2023-11-08T08:19:13.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.157+0000 7f0a52d72700 1 -- start start 2023-11-08T08:19:13.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a52d72700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 0x7f0a4c160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a52d72700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4c0b1810 con 0x7f0a4c15cc80 2023-11-08T08:19:13.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a52d72700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4c0b1af0 con 0x7f0a4c06a270 2023-11-08T08:19:13.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a52d72700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0b2490 con 0x7f0a4c06e570 2023-11-08T08:19:13.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4bfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0a4c15cc80 0x7f0a4c06de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54088/0 (socket says 172.21.15.116:54088) 2023-11-08T08:19:13.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4bfff700 1 -- 172.21.15.116:0/2634380661 learned_addr learned my addr 172.21.15.116:0/2634380661 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:13.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a5130f700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 0x7f0a4c160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949509758 0 0) 0x7f0a4c0b1810 con 0x7f0a4c15cc80 2023-11-08T08:19:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a380035b0 con 0x7f0a4c15cc80 2023-11-08T08:19:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2737388915 0 0) 0x7f0a380035b0 con 0x7f0a4c15cc80 2023-11-08T08:19:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 msgr2=0x7f0a4c160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 0x7f0a4c160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 >> v1:172.21.15.129:6789/0 conn(0x7f0a4c06a270 legacy=0x7f0a4c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0b1dd0 con 0x7f0a4c15cc80 2023-11-08T08:19:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f0a34001670 con 0x7f0a4c15cc80 2023-11-08T08:19:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 >> v1:172.21.15.116:6789/0 conn(0x7f0a4c15cc80 legacy=0x7f0a4c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 0x7f0a38003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38004d90 0x7f0a38005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0b2490 con 0x7f0a38003fb0 2023-11-08T08:19:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3800bba0 con 0x7f0a38004d90 2023-11-08T08:19:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.158+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3800bd60 con 0x7f0a4c06e570 2023-11-08T08:19:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a50b0e700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 0x7f0a38003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a5130f700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38004d90 0x7f0a38005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 -- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 msgr2=0x7f0a38003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c06e570 0x7f0a38003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 -- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38004d90 msgr2=0x7f0a38005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38004d90 0x7f0a38005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 -- 172.21.15.116:0/2634380661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a380035b0 con 0x7f0a38003fb0 2023-11-08T08:19:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a5130f700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38004d90 0x7f0a38005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a38004850 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c270 tx=0x7f0a38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc45185f0777282 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34007c10 con 0x7f0a38003fb0 2023-11-08T08:19:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3400cd90 con 0x7f0a38003fb0 2023-11-08T08:19:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.159+0000 7f0a4b7fe700 1 -- 172.21.15.116:0/2634380661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34005ba0 con 0x7f0a38003fb0 2023-11-08T08:19:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a52d72700 1 -- 172.21.15.116:0/2634380661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 msgr2=0x7f0a38004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a52d72700 1 --2- 172.21.15.116:0/2634380661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a38004850 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c270 tx=0x7f0a38005880 comp rx=0 tx=0).stop 2023-11-08T08:19:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a5130f700 1 -- 172.21.15.116:0/2634380661 reap_dead start 2023-11-08T08:19:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a52d72700 1 -- 172.21.15.116:0/2634380661 shutdown_connections 2023-11-08T08:19:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a52d72700 1 -- 172.21.15.116:0/2634380661 >> 172.21.15.116:0/2634380661 conn(0x7f0a4c06c010 msgr2=0x7f0a4c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a52d72700 1 -- 172.21.15.116:0/2634380661 shutdown_connections 2023-11-08T08:19:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.160+0000 7f0a52d72700 1 -- 172.21.15.116:0/2634380661 wait complete. 2023-11-08T08:19:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.161+0000 7f0a52d72700 1 Processor -- start 2023-11-08T08:19:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.161+0000 7f0a52d72700 1 -- start start 2023-11-08T08:19:13.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a4c15de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c0ae180 0x7f0a4c1e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a4c14dd20 0x7f0a4c15bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0ae940 con 0x7f0a38003fb0 2023-11-08T08:19:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c14e1c0 con 0x7f0a4c14dd20 2023-11-08T08:19:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0ae660 con 0x7f0a4c0ae180 2023-11-08T08:19:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a4c15de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a4c15de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35014/0 (socket says 172.21.15.116:35014) 2023-11-08T08:19:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 -- 172.21.15.116:0/3876615519 learned_addr learned my addr 172.21.15.116:0/3876615519 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c0ae180 0x7f0a4c1e8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a5130f700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a4c14dd20 0x7f0a4c15bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 -- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c0ae180 msgr2=0x7f0a4c1e8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c0ae180 0x7f0a4c1e8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 -- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a4c14dd20 msgr2=0x7f0a4c15bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a4c14dd20 0x7f0a4c15bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 -- 172.21.15.116:0/3876615519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c15daa0 con 0x7f0a38003fb0 2023-11-08T08:19:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a50b0e700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a4c15de20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00f820 tx=0x7f0a3c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0052e0 con 0x7f0a38003fb0 2023-11-08T08:19:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c005440 con 0x7f0a38003fb0 2023-11-08T08:19:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4c0b20b0 con 0x7f0a38003fb0 2023-11-08T08:19:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.162+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0038c0 con 0x7f0a38003fb0 2023-11-08T08:19:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.163+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4c15dc30 con 0x7f0a38003fb0 2023-11-08T08:19:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.163+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4c0ca410 con 0x7f0a38003fb0 2023-11-08T08:19:13.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.164+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a3c003a20 con 0x7f0a38003fb0 2023-11-08T08:19:13.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.164+0000 7f0a497fa700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0a2c068320 0x7f0a2c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.164+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0a2c068320 0x7f0a2c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.165+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0a3c085b60 con 0x7f0a38003fb0 2023-11-08T08:19:13.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.165+0000 7f0a4bfff700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0a2c068320 0x7f0a2c06a7d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0a34000c00 tx=0x7f0a34007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.169+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a4c0ca410 con 0x7f0a38003fb0 2023-11-08T08:19:13.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.349+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a4c0ca410 con 0x7f0a38003fb0 2023-11-08T08:19:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.351+0000 7f0a497fa700 1 -- 172.21.15.116:0/3876615519 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f0a4c0ca410 con 0x7f0a38003fb0 2023-11-08T08:19:13.352 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:13.352 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:13.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0a2c068320 msgr2=0x7f0a2c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0a2c068320 0x7f0a2c06a7d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0a34000c00 tx=0x7f0a34007000 comp rx=0 tx=0).stop 2023-11-08T08:19:13.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 msgr2=0x7f0a4c15de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a4c15de20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00f820 tx=0x7f0a3c00fb30 comp rx=0 tx=0).stop 2023-11-08T08:19:13.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 shutdown_connections 2023-11-08T08:19:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0a2c068320 0x7f0a2c06a7d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a4c14dd20 0x7f0a4c15bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38003fb0 0x7f0a4c15de20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 --2- 172.21.15.116:0/3876615519 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0a4c0ae180 0x7f0a4c1e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 >> 172.21.15.116:0/3876615519 conn(0x7f0a4c06c010 msgr2=0x7f0a4c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:13.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 shutdown_connections 2023-11-08T08:19:13.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.353+0000 7f0a52d72700 1 -- 172.21.15.116:0/3876615519 wait complete. 2023-11-08T08:19:13.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:13.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:13.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.490+0000 7f10cafaa700 1 Processor -- start 2023-11-08T08:19:13.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.490+0000 7f10cafaa700 1 -- start start 2023-11-08T08:19:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10cafaa700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f10c406a980 0x7f10c40ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10cafaa700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10c4071890 0x7f10c4071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10cafaa700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c415cc10 con 0x7f10c40cf710 2023-11-08T08:19:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10cafaa700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c406c9b0 con 0x7f10c4071890 2023-11-08T08:19:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10cafaa700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c406cc90 con 0x7f10c406a980 2023-11-08T08:19:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10c4071890 0x7f10c4071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10c4071890 0x7f10c4071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35018/0 (socket says 172.21.15.116:35018) 2023-11-08T08:19:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 -- 172.21.15.116:0/3283472364 learned_addr learned my addr 172.21.15.116:0/3283472364 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c3fff700 1 --2- 172.21.15.116:0/3283472364 >> v2:172.21.15.129:3300/0 conn(0x7f10c406a980 0x7f10c40ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 -- 172.21.15.116:0/3283472364 >> v2:172.21.15.129:3300/0 conn(0x7f10c406a980 msgr2=0x7f10c40ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 --2- 172.21.15.116:0/3283472364 >> v2:172.21.15.129:3300/0 conn(0x7f10c406a980 0x7f10c40ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 -- 172.21.15.116:0/3283472364 >> v1:172.21.15.129:6789/0 conn(0x7f10c40cf710 legacy=0x7f10c406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 -- 172.21.15.116:0/3283472364 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c406cf70 con 0x7f10c4071890 2023-11-08T08:19:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c9547700 1 --2- 172.21.15.116:0/3283472364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c4071220 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10c40af5e0 tx=0x7f10bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e086bb64d8da50fe server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c37fe700 1 -- 172.21.15.116:0/3283472364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc00e070 con 0x7f10c4071890 2023-11-08T08:19:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c37fe700 1 -- 172.21.15.116:0/3283472364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc00b9c0 con 0x7f10c4071890 2023-11-08T08:19:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.491+0000 7f10c37fe700 1 -- 172.21.15.116:0/3283472364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc00bd70 con 0x7f10c4071890 2023-11-08T08:19:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 -- 172.21.15.116:0/3283472364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 msgr2=0x7f10c4071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 --2- 172.21.15.116:0/3283472364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c4071220 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10c40af5e0 tx=0x7f10bc00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 -- 172.21.15.116:0/3283472364 shutdown_connections 2023-11-08T08:19:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 --2- 172.21.15.116:0/3283472364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c4071220 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 --2- 172.21.15.116:0/3283472364 >> v2:172.21.15.129:3300/0 conn(0x7f10c406a980 0x7f10c40ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 -- 172.21.15.116:0/3283472364 >> 172.21.15.116:0/3283472364 conn(0x7f10c406c010 msgr2=0x7f10c40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 -- 172.21.15.116:0/3283472364 shutdown_connections 2023-11-08T08:19:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.492+0000 7f10cafaa700 1 -- 172.21.15.116:0/3283472364 wait complete. 2023-11-08T08:19:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.493+0000 7f10cafaa700 1 Processor -- start 2023-11-08T08:19:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.493+0000 7f10cafaa700 1 -- start start 2023-11-08T08:19:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10cafaa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c406a980 0x7f10c415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10cafaa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c41f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10cafaa700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10c40cf710 0x7f10c41f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10cafaa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c41f32d0 con 0x7f10c4071890 2023-11-08T08:19:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10cafaa700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c41f2c20 con 0x7f10c406a980 2023-11-08T08:19:13.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10cafaa700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c41f2f00 con 0x7f10c40cf710 2023-11-08T08:19:13.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c41f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c41f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35028/0 (socket says 172.21.15.116:35028) 2023-11-08T08:19:13.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 -- 172.21.15.116:0/492783427 learned_addr learned my addr 172.21.15.116:0/492783427 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:13.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c8d46700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c406a980 0x7f10c415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c9547700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10c40cf710 0x7f10c41f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 -- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10c40cf710 msgr2=0x7f10c41f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10c40cf710 0x7f10c41f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 -- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c406a980 msgr2=0x7f10c415b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c406a980 0x7f10c415b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 -- 172.21.15.116:0/492783427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc009b30 con 0x7f10c4071890 2023-11-08T08:19:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c8d46700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c406a980 0x7f10c415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.494+0000 7f10c3fff700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c41f20e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f10b000e810 tx=0x7f10b000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.495+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0013610 con 0x7f10c4071890 2023-11-08T08:19:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.495+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b0013770 con 0x7f10c4071890 2023-11-08T08:19:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.495+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c406d530 con 0x7f10c4071890 2023-11-08T08:19:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.495+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0013b20 con 0x7f10c4071890 2023-11-08T08:19:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.495+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c406d250 con 0x7f10c4071890 2023-11-08T08:19:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.495+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c4160480 con 0x7f10c4071890 2023-11-08T08:19:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.496+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10b0023070 con 0x7f10c4071890 2023-11-08T08:19:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.496+0000 7f10c17fa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10a80682f0 0x7f10a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.496+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f10b0085a50 con 0x7f10c4071890 2023-11-08T08:19:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.496+0000 7f10c8d46700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10a80682f0 0x7f10a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.497+0000 7f10c8d46700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10a80682f0 0x7f10a806a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f10b8000f60 tx=0x7f10b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.501+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10b0052220 con 0x7f10c4071890 2023-11-08T08:19:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.682+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f10c40ce740 con 0x7f10c4071890 2023-11-08T08:19:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.683+0000 7f10c17fa700 1 -- 172.21.15.116:0/492783427 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f10b00170a0 con 0x7f10c4071890 2023-11-08T08:19:13.684 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.684+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10a80682f0 msgr2=0x7f10a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.684+0000 7f10cafaa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10a80682f0 0x7f10a806a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f10b8000f60 tx=0x7f10b8009040 comp rx=0 tx=0).stop 2023-11-08T08:19:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 msgr2=0x7f10c41f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c41f20e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f10b000e810 tx=0x7f10b000ebd0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 shutdown_connections 2023-11-08T08:19:13.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10c40cf710 0x7f10c41f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10a80682f0 0x7f10a806a7a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c406a980 0x7f10c415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 --2- 172.21.15.116:0/492783427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10c4071890 0x7f10c41f20e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 >> 172.21.15.116:0/492783427 conn(0x7f10c406c010 msgr2=0x7f10c40b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:13.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 shutdown_connections 2023-11-08T08:19:13.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.685+0000 7f10cafaa700 1 -- 172.21.15.116:0/492783427 wait complete. 2023-11-08T08:19:13.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:13.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.819+0000 7fbaddac7700 1 Processor -- start 2023-11-08T08:19:13.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.819+0000 7fbaddac7700 1 -- start start 2023-11-08T08:19:13.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.819+0000 7fbaddac7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbad80ca160 0x7fbad80cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.819+0000 7fbaddac7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad806a310 con 0x7fbad8159710 2023-11-08T08:19:13.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.819+0000 7fbaddac7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad80ce7e0 con 0x7fbad8070370 2023-11-08T08:19:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.819+0000 7fbaddac7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad80707f0 con 0x7fbad80ca160 2023-11-08T08:19:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad77fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbad8159710 0x7fbad80cdff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54098/0 (socket says 172.21.15.116:54098) 2023-11-08T08:19:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad77fe700 1 -- 172.21.15.116:0/583153552 learned_addr learned my addr 172.21.15.116:0/583153552 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600597877 0 0) 0x7fbad806a310 con 0x7fbad8159710 2023-11-08T08:19:13.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 --2- 172.21.15.116:0/583153552 >> v2:172.21.15.129:3300/0 conn(0x7fbad80ca160 0x7fbad80cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac40035b0 con 0x7fbad8159710 2023-11-08T08:19:13.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2455069599 0 0) 0x7fbac40035b0 con 0x7fbad8159710 2023-11-08T08:19:13.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 >> v2:172.21.15.129:3300/0 conn(0x7fbad80ca160 msgr2=0x7fbad80cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 --2- 172.21.15.116:0/583153552 >> v2:172.21.15.129:3300/0 conn(0x7fbad80ca160 0x7fbad80cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 >> v1:172.21.15.129:6789/0 conn(0x7fbad8070370 legacy=0x7fbad80c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad806a5b0 con 0x7fbad8159710 2023-11-08T08:19:13.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fbac0001670 con 0x7fbad8159710 2023-11-08T08:19:13.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 --2- 172.21.15.116:0/583153552 >> v2:172.21.15.129:3300/0 conn(0x7fbad80ca160 0x7fbad80cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:13.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 >> v1:172.21.15.116:6789/0 conn(0x7fbad8159710 legacy=0x7fbad80cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbac4003a00 0x7fbac4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4004850 0x7fbac4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac4005200 0x7fbac400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80707f0 con 0x7fbac4005200 2023-11-08T08:19:13.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac400c170 con 0x7fbac4004850 2023-11-08T08:19:13.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac400c330 con 0x7fbac4003a00 2023-11-08T08:19:13.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac4005200 0x7fbac400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad77fe700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbac4003a00 0x7fbac4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad6ffd700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4004850 0x7fbac4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 -- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbac4003a00 msgr2=0x7fbac4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbac4003a00 0x7fbac4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 -- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4004850 msgr2=0x7fbac4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:13.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.820+0000 7fbad7fff700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4004850 0x7fbac4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad7fff700 1 -- 172.21.15.116:0/583153552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac40035b0 con 0x7fbac4005200 2023-11-08T08:19:13.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad6ffd700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4004850 0x7fbac4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:13.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad77fe700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbac4003a00 0x7fbac4003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:13.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad7fff700 1 -- 172.21.15.116:0/583153552 reap_dead start 2023-11-08T08:19:13.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad7fff700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac4005200 0x7fbac400ba10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbad00073b0 tx=0x7fbac4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b0748f88a8cc6e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0009490 con 0x7fbac4005200 2023-11-08T08:19:13.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad0014400 con 0x7fbac4005200 2023-11-08T08:19:13.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbad67fc700 1 -- 172.21.15.116:0/583153552 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad00147b0 con 0x7fbac4005200 2023-11-08T08:19:13.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 -- 172.21.15.116:0/583153552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac4005200 msgr2=0x7fbac400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac4005200 0x7fbac400ba10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbad00073b0 tx=0x7fbac4005880 comp rx=0 tx=0).stop 2023-11-08T08:19:13.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 -- 172.21.15.116:0/583153552 shutdown_connections 2023-11-08T08:19:13.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 --2- 172.21.15.116:0/583153552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac4005200 0x7fbac400ba10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 -- 172.21.15.116:0/583153552 >> 172.21.15.116:0/583153552 conn(0x7fbad80cf400 msgr2=0x7fbad80cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 -- 172.21.15.116:0/583153552 shutdown_connections 2023-11-08T08:19:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.821+0000 7fbaddac7700 1 -- 172.21.15.116:0/583153552 wait complete. 2023-11-08T08:19:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 Processor -- start 2023-11-08T08:19:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 -- start start 2023-11-08T08:19:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4005200 0x7fbad80b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 0x7fbad80b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbad80c66b0 0x7fbad80c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b2830 con 0x7fbad8158bc0 2023-11-08T08:19:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b2b10 con 0x7fbac4005200 2023-11-08T08:19:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbaddac7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b2df0 con 0x7fbad80c66b0 2023-11-08T08:19:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.823+0000 7fbad6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 0x7fbad80b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 0x7fbad80b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35046/0 (socket says 172.21.15.116:35046) 2023-11-08T08:19:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 -- 172.21.15.116:0/1199286727 learned_addr learned my addr 172.21.15.116:0/1199286727 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad77fe700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4005200 0x7fbad80b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad7fff700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbad80c66b0 0x7fbad80c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 -- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbad80c66b0 msgr2=0x7fbad80c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbad80c66b0 0x7fbad80c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 -- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4005200 msgr2=0x7fbad80b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:13.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4005200 0x7fbad80b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 -- 172.21.15.116:0/1199286727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80ceac0 con 0x7fbad8158bc0 2023-11-08T08:19:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad7fff700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbad80c66b0 0x7fbad80c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad77fe700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4005200 0x7fbad80b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbad6ffd700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 0x7fbad80b3b10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e490 tx=0x7fbac800e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac80052e0 con 0x7fbad8158bc0 2023-11-08T08:19:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8005440 con 0x7fbad8158bc0 2023-11-08T08:19:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac80038c0 con 0x7fbad8158bc0 2023-11-08T08:19:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad806a890 con 0x7fbad8158bc0 2023-11-08T08:19:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.824+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad80cee40 con 0x7fbad8158bc0 2023-11-08T08:19:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.825+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad8159ef0 con 0x7fbad8158bc0 2023-11-08T08:19:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.826+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbac801b030 con 0x7fbad8158bc0 2023-11-08T08:19:13.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.826+0000 7fbacffff700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbab80682f0 0x7fbab806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:13.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.826+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbac80851b0 con 0x7fbad8158bc0 2023-11-08T08:19:13.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.826+0000 7fbad77fe700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbab80682f0 0x7fbab806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:13.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.826+0000 7fbad77fe700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbab80682f0 0x7fbab806a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbac0000fb0 tx=0x7fbac0000f40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:13.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:13.830+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbac804d5e0 con 0x7fbad8158bc0 2023-11-08T08:19:14.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.010+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbad80743f0 con 0x7fbad8158bc0 2023-11-08T08:19:14.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.010+0000 7fbacffff700 1 -- 172.21.15.116:0/1199286727 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbac8019020 con 0x7fbad8158bc0 2023-11-08T08:19:14.012 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:14.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbab80682f0 msgr2=0x7fbab806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbab80682f0 0x7fbab806a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbac0000fb0 tx=0x7fbac0000f40 comp rx=0 tx=0).stop 2023-11-08T08:19:14.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 msgr2=0x7fbad80b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 0x7fbad80b3b10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e490 tx=0x7fbac800e850 comp rx=0 tx=0).stop 2023-11-08T08:19:14.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 shutdown_connections 2023-11-08T08:19:14.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbad80c66b0 0x7fbad80c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbab80682f0 0x7fbab806a7a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac4005200 0x7fbad80b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 --2- 172.21.15.116:0/1199286727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbad8158bc0 0x7fbad80b3b10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.012+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 >> 172.21.15.116:0/1199286727 conn(0x7fbad80cf400 msgr2=0x7fbad8070930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:14.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.013+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 shutdown_connections 2023-11-08T08:19:14.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.013+0000 7fbaddac7700 1 -- 172.21.15.116:0/1199286727 wait complete. 2023-11-08T08:19:14.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.147+0000 7f4d34fb3700 1 Processor -- start 2023-11-08T08:19:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.147+0000 7f4d34fb3700 1 -- start start 2023-11-08T08:19:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.148+0000 7f4d34fb3700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4d300718e0 0x7f4d3006c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.148+0000 7f4d34fb3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d30160a50 con 0x7f4d3015c810 2023-11-08T08:19:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.148+0000 7f4d34fb3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d30160d30 con 0x7f4d3006cdc0 2023-11-08T08:19:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.148+0000 7f4d34fb3700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3015cbe0 con 0x7f4d300718e0 2023-11-08T08:19:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.148+0000 7f4d2e59c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4d3015c810 0x7f4d30160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54110/0 (socket says 172.21.15.116:54110) 2023-11-08T08:19:14.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.148+0000 7f4d2e59c700 1 -- 172.21.15.116:0/2668555100 learned_addr learned my addr 172.21.15.116:0/2668555100 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:14.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321282012 0 0) 0x7f4d30160a50 con 0x7f4d3015c810 2023-11-08T08:19:14.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d180035b0 con 0x7f4d3015c810 2023-11-08T08:19:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2dd9b700 1 --2- 172.21.15.116:0/2668555100 >> v2:172.21.15.129:3300/0 conn(0x7f4d300718e0 0x7f4d3006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2106870808 0 0) 0x7f4d180035b0 con 0x7f4d3015c810 2023-11-08T08:19:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 >> v2:172.21.15.129:3300/0 conn(0x7f4d300718e0 msgr2=0x7f4d3006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 --2- 172.21.15.116:0/2668555100 >> v2:172.21.15.129:3300/0 conn(0x7f4d300718e0 0x7f4d3006c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 >> v1:172.21.15.129:6789/0 conn(0x7f4d3006cdc0 legacy=0x7f4d3006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30161010 con 0x7f4d3015c810 2023-11-08T08:19:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f4d280019f0 con 0x7f4d3015c810 2023-11-08T08:19:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 >> v1:172.21.15.116:6789/0 conn(0x7f4d3015c810 legacy=0x7f4d30160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d18003a30 0x7f4d18003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18004850 0x7f4d18004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d18005200 0x7f4d1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3015cbe0 con 0x7f4d18003a30 2023-11-08T08:19:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1800c0e0 con 0x7f4d18004850 2023-11-08T08:19:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1800c270 con 0x7f4d18005200 2023-11-08T08:19:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d18003a30 0x7f4d18003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2dd9b700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18004850 0x7f4d18004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2ed9d700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d18005200 0x7f4d1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 -- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d18005200 msgr2=0x7f4d1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d18005200 0x7f4d1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 -- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18004850 msgr2=0x7f4d18004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:14.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18004850 0x7f4d18004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 -- 172.21.15.116:0/2668555100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d180035b0 con 0x7f4d18003a30 2023-11-08T08:19:14.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2dd9b700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18004850 0x7f4d18004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:14.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2ed9d700 1 -- 172.21.15.116:0/2668555100 reap_dead start 2023-11-08T08:19:14.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d18003a30 0x7f4d18003e40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800a800 tx=0x7f4d18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c13420af85fe8d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800a2f0 con 0x7f4d18003a30 2023-11-08T08:19:14.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d28012600 con 0x7f4d18003a30 2023-11-08T08:19:14.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.149+0000 7f4d2d59a700 1 -- 172.21.15.116:0/2668555100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d280129b0 con 0x7f4d18003a30 2023-11-08T08:19:14.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 -- 172.21.15.116:0/2668555100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d18003a30 msgr2=0x7f4d18003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d18003a30 0x7f4d18003e40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800a800 tx=0x7f4d18005880 comp rx=0 tx=0).stop 2023-11-08T08:19:14.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 -- 172.21.15.116:0/2668555100 shutdown_connections 2023-11-08T08:19:14.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/2668555100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d18003a30 0x7f4d18003e40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 -- 172.21.15.116:0/2668555100 >> 172.21.15.116:0/2668555100 conn(0x7f4d3006c010 msgr2=0x7f4d300cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:14.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 -- 172.21.15.116:0/2668555100 shutdown_connections 2023-11-08T08:19:14.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.150+0000 7f4d34fb3700 1 -- 172.21.15.116:0/2668555100 wait complete. 2023-11-08T08:19:14.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.151+0000 7f4d34fb3700 1 Processor -- start 2023-11-08T08:19:14.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 -- start start 2023-11-08T08:19:14.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18003a30 0x7f4d3015f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d3015e390 0x7f4d30157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 0x7f4d300b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300b1960 con 0x7f4d300b0f80 2023-11-08T08:19:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3015df00 con 0x7f4d18003a30 2023-11-08T08:19:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d34fb3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300b01a0 con 0x7f4d3015e390 2023-11-08T08:19:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d2ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 0x7f4d300b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d2ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 0x7f4d300b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35066/0 (socket says 172.21.15.116:35066) 2023-11-08T08:19:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d2ed9d700 1 -- 172.21.15.116:0/389730925 learned_addr learned my addr 172.21.15.116:0/389730925 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d2dd9b700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d3015e390 0x7f4d30157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.152+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18003a30 0x7f4d3015f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d2ed9d700 1 -- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d3015e390 msgr2=0x7f4d30157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d2ed9d700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d3015e390 0x7f4d30157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d2ed9d700 1 -- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18003a30 msgr2=0x7f4d3015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d2ed9d700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18003a30 0x7f4d3015f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d2ed9d700 1 -- 172.21.15.116:0/389730925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300ce740 con 0x7f4d300b0f80 2023-11-08T08:19:14.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d2ed9d700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 0x7f4d300b1420 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4d30161320 tx=0x7f4d2000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d200050d0 con 0x7f4d300b0f80 2023-11-08T08:19:14.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d20005230 con 0x7f4d300b0f80 2023-11-08T08:19:14.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20003680 con 0x7f4d300b0f80 2023-11-08T08:19:14.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d300ceb20 con 0x7f4d300b0f80 2023-11-08T08:19:14.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.153+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d300ce8d0 con 0x7f4d300b0f80 2023-11-08T08:19:14.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.154+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3015e090 con 0x7f4d300b0f80 2023-11-08T08:19:14.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.154+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4d2001b030 con 0x7f4d300b0f80 2023-11-08T08:19:14.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.155+0000 7f4d1effd700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4d100682f0 0x7f4d1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.155+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4d20084fc0 con 0x7f4d300b0f80 2023-11-08T08:19:14.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.155+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4d100682f0 0x7f4d1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.155+0000 7f4d2e59c700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4d100682f0 0x7f4d1006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d28000f80 tx=0x7f4d28007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.159+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4d20051730 con 0x7f4d300b0f80 2023-11-08T08:19:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.336+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d3015d6e0 con 0x7f4d300b0f80 2023-11-08T08:19:14.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.337+0000 7f4d1effd700 1 -- 172.21.15.116:0/389730925 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4d20019020 con 0x7f4d300b0f80 2023-11-08T08:19:14.339 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:14.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4d100682f0 msgr2=0x7f4d1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4d100682f0 0x7f4d1006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d28000f80 tx=0x7f4d28007000 comp rx=0 tx=0).stop 2023-11-08T08:19:14.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 msgr2=0x7f4d300b1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 0x7f4d300b1420 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4d30161320 tx=0x7f4d2000e720 comp rx=0 tx=0).stop 2023-11-08T08:19:14.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 shutdown_connections 2023-11-08T08:19:14.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d300b0f80 0x7f4d300b1420 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4d100682f0 0x7f4d1006a7a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d18003a30 0x7f4d3015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 --2- 172.21.15.116:0/389730925 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4d3015e390 0x7f4d30157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 >> 172.21.15.116:0/389730925 conn(0x7f4d3006c010 msgr2=0x7f4d300cc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:14.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 shutdown_connections 2023-11-08T08:19:14.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.339+0000 7f4d34fb3700 1 -- 172.21.15.116:0/389730925 wait complete. 2023-11-08T08:19:14.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:14.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.475+0000 7f181ffde700 1 Processor -- start 2023-11-08T08:19:14.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.475+0000 7f181ffde700 1 -- start start 2023-11-08T08:19:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181ffde700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18180cec00 0x7f1818065fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181ffde700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f18180cac90 0x7f18180cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181ffde700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181815ceb0 con 0x7f1818160300 2023-11-08T08:19:14.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181ffde700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18180cb650 con 0x7f18180cac90 2023-11-08T08:19:14.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181ffde700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18181608c0 con 0x7f18180cec00 2023-11-08T08:19:14.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181e57b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f18180cac90 0x7f18180cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.476+0000 7f181dd7a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18180cec00 0x7f1818065fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f18180cac90 0x7f18180cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:42984/0 (socket says 172.21.15.116:42984) 2023-11-08T08:19:14.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 -- 172.21.15.116:0/434451724 learned_addr learned my addr 172.21.15.116:0/434451724 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:14.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 -- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18180cec00 msgr2=0x7f1818065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:14.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 --2- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18180cec00 0x7f1818065fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 -- 172.21.15.116:0/434451724 >> v1:172.21.15.129:6789/0 conn(0x7f1818160300 legacy=0x7f181815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 -- 172.21.15.116:0/434451724 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818160ba0 con 0x7f18180cac90 2023-11-08T08:19:14.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181dd7a700 1 --2- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18180cec00 0x7f1818065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:14.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181e57b700 1 --2- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18180cb110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1810009e50 tx=0x7f181000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=883d927a63c35e34 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181cd78700 1 -- 172.21.15.116:0/434451724 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181000e070 con 0x7f18180cac90 2023-11-08T08:19:14.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181cd78700 1 -- 172.21.15.116:0/434451724 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181000b910 con 0x7f18180cac90 2023-11-08T08:19:14.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.477+0000 7f181cd78700 1 -- 172.21.15.116:0/434451724 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181000bcc0 con 0x7f18180cac90 2023-11-08T08:19:14.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 -- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 msgr2=0x7f18180cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 --2- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18180cb110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1810009e50 tx=0x7f181000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:14.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 -- 172.21.15.116:0/434451724 shutdown_connections 2023-11-08T08:19:14.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 --2- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18180cb110 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 --2- 172.21.15.116:0/434451724 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18180cec00 0x7f1818065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 -- 172.21.15.116:0/434451724 >> 172.21.15.116:0/434451724 conn(0x7f18180cf400 msgr2=0x7f18180cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 -- 172.21.15.116:0/434451724 shutdown_connections 2023-11-08T08:19:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.478+0000 7f181ffde700 1 -- 172.21.15.116:0/434451724 wait complete. 2023-11-08T08:19:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.479+0000 7f181ffde700 1 Processor -- start 2023-11-08T08:19:14.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.479+0000 7f181ffde700 1 -- start start 2023-11-08T08:19:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181ffde700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18181529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181ffde700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 0x7f18180b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181ffde700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1818160300 0x7f181815f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181ffde700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180ad8e0 con 0x7f18180cec00 2023-11-08T08:19:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181ffde700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180b3d00 con 0x7f18180cac90 2023-11-08T08:19:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181ffde700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818152f10 con 0x7f1818160300 2023-11-08T08:19:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 0x7f18180b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 0x7f18180b05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35076/0 (socket says 172.21.15.116:35076) 2023-11-08T08:19:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 -- 172.21.15.116:0/3991012742 learned_addr learned my addr 172.21.15.116:0/3991012742 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181dd7a700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18181529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181e57b700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1818160300 0x7f181815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 -- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1818160300 msgr2=0x7f181815f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1818160300 0x7f181815f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 -- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 msgr2=0x7f18181529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18181529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.480+0000 7f181d579700 1 -- 172.21.15.116:0/3991012742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18080094a0 con 0x7f18180cec00 2023-11-08T08:19:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.481+0000 7f181d579700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 0x7f18180b05f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f181400c450 tx=0x7f181400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.481+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18140052d0 con 0x7f18180cec00 2023-11-08T08:19:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.481+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814005430 con 0x7f18180cec00 2023-11-08T08:19:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.481+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18140038c0 con 0x7f18180cec00 2023-11-08T08:19:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.481+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1810009b30 con 0x7f18180cec00 2023-11-08T08:19:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.481+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1818161160 con 0x7f18180cec00 2023-11-08T08:19:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.483+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1814019030 con 0x7f18180cec00 2023-11-08T08:19:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.483+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1818160e80 con 0x7f18180cec00 2023-11-08T08:19:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.483+0000 7f180e7fc700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17fc0682f0 0x7f17fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.483+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1814084e10 con 0x7f18180cec00 2023-11-08T08:19:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.483+0000 7f181dd7a700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17fc0682f0 0x7f17fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.483+0000 7f181dd7a700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17fc0682f0 0x7f17fc06a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1808009d70 tx=0x7f1808009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.489+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1814051670 con 0x7f18180cec00 2023-11-08T08:19:14.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.672+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18180c98a0 con 0x7f18180cec00 2023-11-08T08:19:14.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.673+0000 7f180e7fc700 1 -- 172.21.15.116:0/3991012742 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18140170a0 con 0x7f18180cec00 2023-11-08T08:19:14.675 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:14.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17fc0682f0 msgr2=0x7f17fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17fc0682f0 0x7f17fc06a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1808009d70 tx=0x7f1808009210 comp rx=0 tx=0).stop 2023-11-08T08:19:14.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 msgr2=0x7f18180b05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 0x7f18180b05f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f181400c450 tx=0x7f181400c760 comp rx=0 tx=0).stop 2023-11-08T08:19:14.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 shutdown_connections 2023-11-08T08:19:14.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1818160300 0x7f181815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17fc0682f0 0x7f17fc06a7a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18180cac90 0x7f18181529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 --2- 172.21.15.116:0/3991012742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18180cec00 0x7f18180b05f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 >> 172.21.15.116:0/3991012742 conn(0x7f18180cf400 msgr2=0x7f181806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:14.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 shutdown_connections 2023-11-08T08:19:14.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.675+0000 7f181ffde700 1 -- 172.21.15.116:0/3991012742 wait complete. 2023-11-08T08:19:14.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.808+0000 7f24da4f2700 1 Processor -- start 2023-11-08T08:19:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.808+0000 7f24da4f2700 1 -- start start 2023-11-08T08:19:14.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24da4f2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f24d415fe50 0x7f24d406d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24da4f2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24d40b3b80 0x7f24d4160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24da4f2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4161370 con 0x7f24d40b27d0 2023-11-08T08:19:14.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24da4f2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d4160220 con 0x7f24d40b3b80 2023-11-08T08:19:14.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24da4f2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d40cf700 con 0x7f24d415fe50 2023-11-08T08:19:14.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24d40b3b80 0x7f24d4160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24d40b3b80 0x7f24d4160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35082/0 (socket says 172.21.15.116:35082) 2023-11-08T08:19:14.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 -- 172.21.15.116:0/1767862131 learned_addr learned my addr 172.21.15.116:0/1767862131 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:14.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d3fff700 1 --2- 172.21.15.116:0/1767862131 >> v2:172.21.15.129:3300/0 conn(0x7f24d415fe50 0x7f24d406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 -- 172.21.15.116:0/1767862131 >> v2:172.21.15.129:3300/0 conn(0x7f24d415fe50 msgr2=0x7f24d406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:14.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 --2- 172.21.15.116:0/1767862131 >> v2:172.21.15.129:3300/0 conn(0x7f24d415fe50 0x7f24d406d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 -- 172.21.15.116:0/1767862131 >> v1:172.21.15.129:6789/0 conn(0x7f24d40b27d0 legacy=0x7f24d40cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:14.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.809+0000 7f24d37fe700 1 -- 172.21.15.116:0/1767862131 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b3fd0 con 0x7f24d40b3b80 2023-11-08T08:19:14.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24d37fe700 1 --2- 172.21.15.116:0/1767862131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b3b80 0x7f24d4160ad0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f24d40d1e30 tx=0x7f24c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b0cc3f5ba0464e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24d2ffd700 1 -- 172.21.15.116:0/1767862131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c4004230 con 0x7f24d40b3b80 2023-11-08T08:19:14.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24d2ffd700 1 -- 172.21.15.116:0/1767862131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c4004390 con 0x7f24d40b3b80 2023-11-08T08:19:14.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24d2ffd700 1 -- 172.21.15.116:0/1767862131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c40036a0 con 0x7f24d40b3b80 2023-11-08T08:19:14.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 -- 172.21.15.116:0/1767862131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b3b80 msgr2=0x7f24d4160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 --2- 172.21.15.116:0/1767862131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b3b80 0x7f24d4160ad0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f24d40d1e30 tx=0x7f24c4009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 -- 172.21.15.116:0/1767862131 shutdown_connections 2023-11-08T08:19:14.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 --2- 172.21.15.116:0/1767862131 >> v2:172.21.15.129:3300/0 conn(0x7f24d415fe50 0x7f24d406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 --2- 172.21.15.116:0/1767862131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b3b80 0x7f24d4160ad0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 -- 172.21.15.116:0/1767862131 >> 172.21.15.116:0/1767862131 conn(0x7f24d406ea80 msgr2=0x7f24d406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:14.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 -- 172.21.15.116:0/1767862131 shutdown_connections 2023-11-08T08:19:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.810+0000 7f24da4f2700 1 -- 172.21.15.116:0/1767862131 wait complete. 2023-11-08T08:19:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.811+0000 7f24da4f2700 1 Processor -- start 2023-11-08T08:19:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 -- start start 2023-11-08T08:19:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 0x7f24d415b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24d40b3b80 0x7f24d40c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24d415fe50 0x7f24d40c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40c74d0 con 0x7f24d40b27d0 2023-11-08T08:19:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415b590 con 0x7f24d415fe50 2023-11-08T08:19:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24da4f2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415ad10 con 0x7f24d40b3b80 2023-11-08T08:19:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24d3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 0x7f24d415b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24d3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 0x7f24d415b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35088/0 (socket says 172.21.15.116:35088) 2023-11-08T08:19:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.812+0000 7f24d3fff700 1 -- 172.21.15.116:0/3772212241 learned_addr learned my addr 172.21.15.116:0/3772212241 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d8a8f700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24d415fe50 0x7f24d40c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d37fe700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24d40b3b80 0x7f24d40c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d3fff700 1 -- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24d40b3b80 msgr2=0x7f24d40c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d3fff700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24d40b3b80 0x7f24d40c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d3fff700 1 -- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24d415fe50 msgr2=0x7f24d40c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d3fff700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24d415fe50 0x7f24d40c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d3fff700 1 -- 172.21.15.116:0/3772212241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c4009550 con 0x7f24d40b27d0 2023-11-08T08:19:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d3fff700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 0x7f24d415b050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24c800e810 tx=0x7f24c800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8013610 con 0x7f24d40b27d0 2023-11-08T08:19:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c8013770 con 0x7f24d40b27d0 2023-11-08T08:19:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d40cf0b0 con 0x7f24d40b27d0 2023-11-08T08:19:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8013b20 con 0x7f24d40b27d0 2023-11-08T08:19:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.813+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d40cedd0 con 0x7f24d40b27d0 2023-11-08T08:19:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.814+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d40cf390 con 0x7f24d40b27d0 2023-11-08T08:19:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.814+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24c8023070 con 0x7f24d40b27d0 2023-11-08T08:19:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.815+0000 7f24d0ff9700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24c00682f0 0x7f24c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.815+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f24c8085970 con 0x7f24d40b27d0 2023-11-08T08:19:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.815+0000 7f24d37fe700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24c00682f0 0x7f24c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.815+0000 7f24d37fe700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24c00682f0 0x7f24c006a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009130 tx=0x7f24c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:14.819+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24c8052150 con 0x7f24d40b27d0 2023-11-08T08:19:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.021+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f24d4160410 con 0x7f24d40b27d0 2023-11-08T08:19:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.021+0000 7f24d0ff9700 1 -- 172.21.15.116:0/3772212241 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f24c8055770 con 0x7f24d40b27d0 2023-11-08T08:19:15.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24c00682f0 msgr2=0x7f24c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24c00682f0 0x7f24c006a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009130 tx=0x7f24c4008030 comp rx=0 tx=0).stop 2023-11-08T08:19:15.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 msgr2=0x7f24d415b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 0x7f24d415b050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24c800e810 tx=0x7f24c800ebd0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 shutdown_connections 2023-11-08T08:19:15.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24c00682f0 0x7f24c006a7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24d415fe50 0x7f24d40c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24d40b27d0 0x7f24d415b050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 --2- 172.21.15.116:0/3772212241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24d40b3b80 0x7f24d40c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 >> 172.21.15.116:0/3772212241 conn(0x7f24d406ea80 msgr2=0x7f24d40b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:15.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 shutdown_connections 2023-11-08T08:19:15.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.023+0000 7f24da4f2700 1 -- 172.21.15.116:0/3772212241 wait complete. 2023-11-08T08:19:15.027 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:15.036 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:15.036 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:15.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 Processor -- start 2023-11-08T08:19:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 -- start start 2023-11-08T08:19:15.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f58dc0d0c60 0x7f58dc0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc06e400 0x7f58dc0d1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0d1570 con 0x7f58dc160e70 2023-11-08T08:19:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc06e900 con 0x7f58dc0d0c60 2023-11-08T08:19:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.215+0000 7f58e14a0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc160390 con 0x7f58dc06e400 2023-11-08T08:19:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58daffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f58dc0d0c60 0x7f58dc0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58da7fc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc06e400 0x7f58dc0d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58db7fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f58dc160e70 0x7f58dc06f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:51206/0 (socket says 172.21.15.116:51206) 2023-11-08T08:19:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58db7fe700 1 -- 172.21.15.116:0/694787168 learned_addr learned my addr 172.21.15.116:0/694787168 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58d9ffb700 1 -- 172.21.15.116:0/694787168 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863123954 0 0) 0x7f58dc0d1570 con 0x7f58dc160e70 2023-11-08T08:19:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58d9ffb700 1 -- 172.21.15.116:0/694787168 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c40035b0 con 0x7f58dc160e70 2023-11-08T08:19:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58da7fc700 1 -- 172.21.15.116:0/694787168 >> v2:172.21.15.129:3300/0 conn(0x7f58dc0d0c60 msgr2=0x7f58dc0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58da7fc700 1 --2- 172.21.15.116:0/694787168 >> v2:172.21.15.129:3300/0 conn(0x7f58dc0d0c60 0x7f58dc0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58da7fc700 1 -- 172.21.15.116:0/694787168 >> v1:172.21.15.129:6789/0 conn(0x7f58dc160e70 legacy=0x7f58dc06f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58da7fc700 1 -- 172.21.15.116:0/694787168 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc160670 con 0x7f58dc06e400 2023-11-08T08:19:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58daffd700 1 --2- 172.21.15.116:0/694787168 >> v2:172.21.15.129:3300/0 conn(0x7f58dc0d0c60 0x7f58dc0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58da7fc700 1 --2- 172.21.15.116:0/694787168 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc06e400 0x7f58dc0d1030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009840 tx=0x7f58d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7b291c0da48917 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.216+0000 7f58d9ffb700 1 -- 172.21.15.116:0/694787168 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0011830 con 0x7f58dc06e400 2023-11-08T08:19:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58d9ffb700 1 -- 172.21.15.116:0/694787168 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d0011990 con 0x7f58dc06e400 2023-11-08T08:19:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58d9ffb700 1 -- 172.21.15.116:0/694787168 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0011d20 con 0x7f58dc06e400 2023-11-08T08:19:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 -- 172.21.15.116:0/694787168 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc06e400 msgr2=0x7f58dc0d1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 --2- 172.21.15.116:0/694787168 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc06e400 0x7f58dc0d1030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009840 tx=0x7f58d0009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 -- 172.21.15.116:0/694787168 shutdown_connections 2023-11-08T08:19:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 --2- 172.21.15.116:0/694787168 >> v2:172.21.15.129:3300/0 conn(0x7f58dc0d0c60 0x7f58dc0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 --2- 172.21.15.116:0/694787168 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc06e400 0x7f58dc0d1030 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 -- 172.21.15.116:0/694787168 >> 172.21.15.116:0/694787168 conn(0x7f58dc06a270 msgr2=0x7f58dc06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 -- 172.21.15.116:0/694787168 shutdown_connections 2023-11-08T08:19:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.217+0000 7f58e14a0700 1 -- 172.21.15.116:0/694787168 wait complete. 2023-11-08T08:19:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.218+0000 7f58e14a0700 1 Processor -- start 2023-11-08T08:19:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.218+0000 7f58e14a0700 1 -- start start 2023-11-08T08:19:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58e14a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 0x7f58dc14c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58e14a0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58dc0d0c60 0x7f58dc154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58e14a0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc160e70 0x7f58dc0b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58e14a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0b1580 con 0x7f58dc06e400 2023-11-08T08:19:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58e14a0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0b0970 con 0x7f58dc0d0c60 2023-11-08T08:19:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58e14a0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0b0c50 con 0x7f58dc160e70 2023-11-08T08:19:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58daffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 0x7f58dc14c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58daffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 0x7f58dc14c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35096/0 (socket says 172.21.15.116:35096) 2023-11-08T08:19:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58daffd700 1 -- 172.21.15.116:0/334122843 learned_addr learned my addr 172.21.15.116:0/334122843 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58db7fe700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc160e70 0x7f58dc0b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.219+0000 7f58da7fc700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58dc0d0c60 0x7f58dc154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58daffd700 1 -- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc160e70 msgr2=0x7f58dc0b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58daffd700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc160e70 0x7f58dc0b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58daffd700 1 -- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58dc0d0c60 msgr2=0x7f58dc154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58daffd700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58dc0d0c60 0x7f58dc154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58daffd700 1 -- 172.21.15.116:0/334122843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc0094a0 con 0x7f58dc06e400 2023-11-08T08:19:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58da7fc700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58dc0d0c60 0x7f58dc154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58daffd700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 0x7f58dc14c320 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00ebe0 tx=0x7f58cc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc009800 con 0x7f58dc06e400 2023-11-08T08:19:15.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc00c910 con 0x7f58dc06e400 2023-11-08T08:19:15.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc00cc80 con 0x7f58dc06e400 2023-11-08T08:19:15.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d00094a0 con 0x7f58dc06e400 2023-11-08T08:19:15.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.220+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc0d0740 con 0x7f58dc06e400 2023-11-08T08:19:15.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.221+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc160950 con 0x7f58dc06e400 2023-11-08T08:19:15.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.221+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f58cc019030 con 0x7f58dc06e400 2023-11-08T08:19:15.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.222+0000 7f58c37fe700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f58c80682f0 0x7f58c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.222+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f58cc085c70 con 0x7f58dc06e400 2023-11-08T08:19:15.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.222+0000 7f58da7fc700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f58c80682f0 0x7f58c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.222+0000 7f58da7fc700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f58c80682f0 0x7f58c806a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f58dc06e830 tx=0x7f58d001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.226+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58cc0524d0 con 0x7f58dc06e400 2023-11-08T08:19:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.407+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58dc15d980 con 0x7f58dc06e400 2023-11-08T08:19:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.408+0000 7f58c37fe700 1 -- 172.21.15.116:0/334122843 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f58cc017090 con 0x7f58dc06e400 2023-11-08T08:19:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:15.411 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:15.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f58c80682f0 msgr2=0x7f58c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f58c80682f0 0x7f58c806a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f58dc06e830 tx=0x7f58d001c040 comp rx=0 tx=0).stop 2023-11-08T08:19:15.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 msgr2=0x7f58dc14c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 0x7f58dc14c320 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00ebe0 tx=0x7f58cc00eef0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 shutdown_connections 2023-11-08T08:19:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f58dc160e70 0x7f58dc0b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc06e400 0x7f58dc14c320 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f58c80682f0 0x7f58c806a7a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 --2- 172.21.15.116:0/334122843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58dc0d0c60 0x7f58dc154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.410+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 >> 172.21.15.116:0/334122843 conn(0x7f58dc06a270 msgr2=0x7f58dc15a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.411+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 shutdown_connections 2023-11-08T08:19:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.411+0000 7f58e14a0700 1 -- 172.21.15.116:0/334122843 wait complete. 2023-11-08T08:19:15.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:15.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:15.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.551+0000 7efde8233700 1 Processor -- start 2023-11-08T08:19:15.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.551+0000 7efde8233700 1 -- start start 2023-11-08T08:19:15.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde8233700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde0159710 0x7efde0070370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde8233700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efde00cee40 0x7efde00cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde8233700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efde00c9a50 0x7efde00c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde8233700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00cf2b0 con 0x7efde00cee40 2023-11-08T08:19:15.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde8233700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00ca410 con 0x7efde00c9a50 2023-11-08T08:19:15.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde8233700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde006a310 con 0x7efde0159710 2023-11-08T08:19:15.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efde00cee40 0x7efde00cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efde00cee40 0x7efde00cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35098/0 (socket says 172.21.15.116:35098) 2023-11-08T08:19:15.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 -- 172.21.15.116:0/809381751 learned_addr learned my addr 172.21.15.116:0/809381751 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:15.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde67d0700 1 --2- 172.21.15.116:0/809381751 >> v2:172.21.15.129:3300/0 conn(0x7efde00c9a50 0x7efde00c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde5fcf700 1 --2- 172.21.15.116:0/809381751 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde0159710 0x7efde0070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 -- 172.21.15.116:0/809381751 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde0159710 msgr2=0x7efde0070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:15.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 --2- 172.21.15.116:0/809381751 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde0159710 0x7efde0070370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 -- 172.21.15.116:0/809381751 >> v2:172.21.15.129:3300/0 conn(0x7efde00c9a50 msgr2=0x7efde00c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 --2- 172.21.15.116:0/809381751 >> v2:172.21.15.129:3300/0 conn(0x7efde00c9a50 0x7efde00c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 -- 172.21.15.116:0/809381751 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00ce530 con 0x7efde00cee40 2023-11-08T08:19:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde57ce700 1 --2- 172.21.15.116:0/809381751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00cdff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efde006d7d0 tx=0x7efddc00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c9f72043d8d50a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde4fcd700 1 -- 172.21.15.116:0/809381751 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc00dd30 con 0x7efde00cee40 2023-11-08T08:19:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde4fcd700 1 -- 172.21.15.116:0/809381751 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc00de90 con 0x7efde00cee40 2023-11-08T08:19:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.552+0000 7efde4fcd700 1 -- 172.21.15.116:0/809381751 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc010690 con 0x7efde00cee40 2023-11-08T08:19:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 -- 172.21.15.116:0/809381751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 msgr2=0x7efde00cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 --2- 172.21.15.116:0/809381751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00cdff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efde006d7d0 tx=0x7efddc00be70 comp rx=0 tx=0).stop 2023-11-08T08:19:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 -- 172.21.15.116:0/809381751 shutdown_connections 2023-11-08T08:19:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 --2- 172.21.15.116:0/809381751 >> v2:172.21.15.129:3300/0 conn(0x7efde00c9a50 0x7efde00c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 --2- 172.21.15.116:0/809381751 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde0159710 0x7efde0070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 --2- 172.21.15.116:0/809381751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00cdff0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 -- 172.21.15.116:0/809381751 >> 172.21.15.116:0/809381751 conn(0x7efde00cf400 msgr2=0x7efde00cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 -- 172.21.15.116:0/809381751 shutdown_connections 2023-11-08T08:19:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.553+0000 7efde8233700 1 -- 172.21.15.116:0/809381751 wait complete. 2023-11-08T08:19:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.554+0000 7efde8233700 1 Processor -- start 2023-11-08T08:19:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.554+0000 7efde8233700 1 -- start start 2023-11-08T08:19:15.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde8233700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde00c9a50 0x7efde014a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde8233700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde8233700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efde0159710 0x7efde00c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde8233700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c8580 con 0x7efde00cee40 2023-11-08T08:19:15.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde8233700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b2460 con 0x7efde0159710 2023-11-08T08:19:15.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde8233700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde014ad90 con 0x7efde00c9a50 2023-11-08T08:19:15.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35108/0 (socket says 172.21.15.116:35108) 2023-11-08T08:19:15.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 -- 172.21.15.116:0/498054104 learned_addr learned my addr 172.21.15.116:0/498054104 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:15.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde67d0700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efde0159710 0x7efde00c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 -- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde00c9a50 msgr2=0x7efde014a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde00c9a50 0x7efde014a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 -- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efde0159710 msgr2=0x7efde00c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efde0159710 0x7efde00c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 -- 172.21.15.116:0/498054104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc00b7f0 con 0x7efde00cee40 2023-11-08T08:19:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.555+0000 7efde57ce700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00c6f60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efde006beb0 tx=0x7efddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.556+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc01f070 con 0x7efde00cee40 2023-11-08T08:19:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.556+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc003bd0 con 0x7efde00cee40 2023-11-08T08:19:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.556+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde006a5b0 con 0x7efde00cee40 2023-11-08T08:19:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.556+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc010470 con 0x7efde00cee40 2023-11-08T08:19:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.556+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde00ce7e0 con 0x7efde00cee40 2023-11-08T08:19:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.557+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde006a870 con 0x7efde00cee40 2023-11-08T08:19:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.557+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efddc02d430 con 0x7efde00cee40 2023-11-08T08:19:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.557+0000 7efdd67fc700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efdc40682f0 0x7efdc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.557+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efddc089b60 con 0x7efde00cee40 2023-11-08T08:19:15.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.557+0000 7efde5fcf700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efdc40682f0 0x7efdc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.558+0000 7efde5fcf700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efdc40682f0 0x7efdc406a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efdd0006ba0 tx=0x7efdd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.562+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efddc056270 con 0x7efde00cee40 2023-11-08T08:19:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.741+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efde00b2610 con 0x7efde00cee40 2023-11-08T08:19:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.741+0000 7efdd67fc700 1 -- 172.21.15.116:0/498054104 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7efde00b2610 con 0x7efde00cee40 2023-11-08T08:19:15.743 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:15.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efdc40682f0 msgr2=0x7efdc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efdc40682f0 0x7efdc406a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efdd0006ba0 tx=0x7efdd0008040 comp rx=0 tx=0).stop 2023-11-08T08:19:15.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 msgr2=0x7efde00c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00c6f60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efde006beb0 tx=0x7efddc004750 comp rx=0 tx=0).stop 2023-11-08T08:19:15.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 shutdown_connections 2023-11-08T08:19:15.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efdc40682f0 0x7efdc406a7a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efde0159710 0x7efde00c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efde00c9a50 0x7efde014a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 --2- 172.21.15.116:0/498054104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde00cee40 0x7efde00c6f60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 >> 172.21.15.116:0/498054104 conn(0x7efde00cf400 msgr2=0x7efde00aef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:15.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 shutdown_connections 2023-11-08T08:19:15.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.743+0000 7efde8233700 1 -- 172.21.15.116:0/498054104 wait complete. 2023-11-08T08:19:15.756 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.877+0000 7f4eab93b700 1 Processor -- start 2023-11-08T08:19:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.877+0000 7f4eab93b700 1 -- start start 2023-11-08T08:19:15.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4eab93b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ea415c810 0x7f4ea4160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4eab93b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4ea406a270 0x7f4ea406a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4eab93b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea406abb0 con 0x7f4ea40718e0 2023-11-08T08:19:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4eab93b700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea415cbe0 con 0x7f4ea415c810 2023-11-08T08:19:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4eab93b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea41607a0 con 0x7f4ea406a270 2023-11-08T08:19:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4ea96d7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ea415c810 0x7f4ea4160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4ea96d7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ea415c810 0x7f4ea4160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35116/0 (socket says 172.21.15.116:35116) 2023-11-08T08:19:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.878+0000 7f4ea96d7700 1 -- 172.21.15.116:0/16006739 learned_addr learned my addr 172.21.15.116:0/16006739 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4ea9ed8700 1 --2- 172.21.15.116:0/16006739 >> v2:172.21.15.129:3300/0 conn(0x7f4ea406a270 0x7f4ea406a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4ea96d7700 1 -- 172.21.15.116:0/16006739 >> v2:172.21.15.129:3300/0 conn(0x7f4ea406a270 msgr2=0x7f4ea406a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4ea96d7700 1 --2- 172.21.15.116:0/16006739 >> v2:172.21.15.129:3300/0 conn(0x7f4ea406a270 0x7f4ea406a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4ea96d7700 1 -- 172.21.15.116:0/16006739 >> v1:172.21.15.129:6789/0 conn(0x7f4ea40718e0 legacy=0x7f4ea406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4ea96d7700 1 -- 172.21.15.116:0/16006739 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4160a50 con 0x7f4ea415c810 2023-11-08T08:19:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4ea96d7700 1 --2- 172.21.15.116:0/16006739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea415c810 0x7f4ea4160260 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4071870 tx=0x7f4e94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8a25003ce00ba6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4e9bfff700 1 -- 172.21.15.116:0/16006739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94004230 con 0x7f4ea415c810 2023-11-08T08:19:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4e9bfff700 1 -- 172.21.15.116:0/16006739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94004390 con 0x7f4ea415c810 2023-11-08T08:19:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4e9bfff700 1 -- 172.21.15.116:0/16006739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e940036a0 con 0x7f4ea415c810 2023-11-08T08:19:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4eab93b700 1 -- 172.21.15.116:0/16006739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea415c810 msgr2=0x7f4ea4160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.879+0000 7f4eab93b700 1 --2- 172.21.15.116:0/16006739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea415c810 0x7f4ea4160260 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4071870 tx=0x7f4e94009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.880+0000 7f4eab93b700 1 -- 172.21.15.116:0/16006739 shutdown_connections 2023-11-08T08:19:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.880+0000 7f4eab93b700 1 --2- 172.21.15.116:0/16006739 >> v2:172.21.15.129:3300/0 conn(0x7f4ea406a270 0x7f4ea406a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.880+0000 7f4eab93b700 1 --2- 172.21.15.116:0/16006739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea415c810 0x7f4ea4160260 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.880+0000 7f4eab93b700 1 -- 172.21.15.116:0/16006739 >> 172.21.15.116:0/16006739 conn(0x7f4ea406c010 msgr2=0x7f4ea40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:15.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.880+0000 7f4eab93b700 1 -- 172.21.15.116:0/16006739 shutdown_connections 2023-11-08T08:19:15.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.880+0000 7f4eab93b700 1 -- 172.21.15.116:0/16006739 wait complete. 2023-11-08T08:19:15.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 Processor -- start 2023-11-08T08:19:15.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 -- start start 2023-11-08T08:19:15.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 0x7f4ea40c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ea40718e0 0x7f4ea4147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ea415c810 0x7f4ea40b1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40b21c0 con 0x7f4ea406a270 2023-11-08T08:19:15.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.881+0000 7f4eab93b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea415de50 con 0x7f4ea40718e0 2023-11-08T08:19:15.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4eab93b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea415e130 con 0x7f4ea415c810 2023-11-08T08:19:15.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 0x7f4ea40c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 0x7f4ea40c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35132/0 (socket says 172.21.15.116:35132) 2023-11-08T08:19:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 -- 172.21.15.116:0/3534231675 learned_addr learned my addr 172.21.15.116:0/3534231675 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea9ed8700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ea415c810 0x7f4ea40b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea8ed6700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ea40718e0 0x7f4ea4147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 -- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ea415c810 msgr2=0x7f4ea40b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ea415c810 0x7f4ea40b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 -- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ea40718e0 msgr2=0x7f4ea4147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ea40718e0 0x7f4ea4147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 -- 172.21.15.116:0/3534231675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94009550 con 0x7f4ea406a270 2023-11-08T08:19:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4ea96d7700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 0x7f4ea40c96c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009130 tx=0x7f4e94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94011630 con 0x7f4ea406a270 2023-11-08T08:19:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94011790 con 0x7f4ea406a270 2023-11-08T08:19:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea4161010 con 0x7f4ea406a270 2023-11-08T08:19:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94011b40 con 0x7f4ea406a270 2023-11-08T08:19:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.882+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea4160d30 con 0x7f4ea406a270 2023-11-08T08:19:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.883+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea40ceb20 con 0x7f4ea406a270 2023-11-08T08:19:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.884+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4e94017020 con 0x7f4ea406a270 2023-11-08T08:19:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.884+0000 7f4e99ffb700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4e880682f0 0x7f4e8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.884+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4e94087110 con 0x7f4ea406a270 2023-11-08T08:19:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.884+0000 7f4ea8ed6700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4e880682f0 0x7f4e8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.884+0000 7f4ea8ed6700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4e880682f0 0x7f4e8806a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ea40b4e90 tx=0x7f4ea0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:15.888+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4e94053970 con 0x7f4ea406a270 2023-11-08T08:19:16.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.067+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ea40743f0 con 0x7f4ea406a270 2023-11-08T08:19:16.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.068+0000 7f4e99ffb700 1 -- 172.21.15.116:0/3534231675 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e94056f90 con 0x7f4ea406a270 2023-11-08T08:19:16.070 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.069+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4e880682f0 msgr2=0x7f4e8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.069+0000 7f4eab93b700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4e880682f0 0x7f4e8806a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ea40b4e90 tx=0x7f4ea0009040 comp rx=0 tx=0).stop 2023-11-08T08:19:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.069+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 msgr2=0x7f4ea40c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 0x7f4ea40c96c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009130 tx=0x7f4e94004750 comp rx=0 tx=0).stop 2023-11-08T08:19:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 shutdown_connections 2023-11-08T08:19:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ea415c810 0x7f4ea40b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea406a270 0x7f4ea40c96c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4e880682f0 0x7f4e8806a7a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 --2- 172.21.15.116:0/3534231675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ea40718e0 0x7f4ea4147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 >> 172.21.15.116:0/3534231675 conn(0x7f4ea406c010 msgr2=0x7f4ea40d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 shutdown_connections 2023-11-08T08:19:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.070+0000 7f4eab93b700 1 -- 172.21.15.116:0/3534231675 wait complete. 2023-11-08T08:19:16.083 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.206+0000 7f13c6b11700 1 Processor -- start 2023-11-08T08:19:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.206+0000 7f13c6b11700 1 -- start start 2023-11-08T08:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c6b11700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 0x7f13c00ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c6b11700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f13c0071890 0x7f13c0071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c6b11700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c015cc10 con 0x7f13c00cf710 2023-11-08T08:19:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c6b11700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c006c9b0 con 0x7f13c0071890 2023-11-08T08:19:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c6b11700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c006cc90 con 0x7f13c006a980 2023-11-08T08:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c48ad700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13c00cf710 0x7f13c006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54112/0 (socket says 172.21.15.116:54112) 2023-11-08T08:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c48ad700 1 -- 172.21.15.116:0/602547716 learned_addr learned my addr 172.21.15.116:0/602547716 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756137282 0 0) 0x7f13c015cc10 con 0x7f13c00cf710 2023-11-08T08:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13c50ae700 1 --2- 172.21.15.116:0/602547716 >> v2:172.21.15.129:3300/0 conn(0x7f13c0071890 0x7f13c0071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b00035b0 con 0x7f13c00cf710 2023-11-08T08:19:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bffff700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 0x7f13c00ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388625449 0 0) 0x7f13b00035b0 con 0x7f13c00cf710 2023-11-08T08:19:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 msgr2=0x7f13c00ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 0x7f13c00ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 >> v2:172.21.15.129:3300/0 conn(0x7f13c0071890 msgr2=0x7f13c0071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 --2- 172.21.15.116:0/602547716 >> v2:172.21.15.129:3300/0 conn(0x7f13c0071890 0x7f13c0071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.207+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c006cf70 con 0x7f13c00cf710 2023-11-08T08:19:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f13b4001670 con 0x7f13c00cf710 2023-11-08T08:19:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 >> v1:172.21.15.116:6789/0 conn(0x7f13c00cf710 legacy=0x7f13c006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13b0003a00 0x7f13b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 0x7f13b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13b0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c006c9b0 con 0x7f13b0004d90 2023-11-08T08:19:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c006cc90 con 0x7f13b0003a00 2023-11-08T08:19:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b000ba10 con 0x7f13c006a980 2023-11-08T08:19:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13b0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bffff700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 0x7f13b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c48ad700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13b0003a00 0x7f13b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 -- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 msgr2=0x7f13b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c006a980 0x7f13b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 -- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13b0003a00 msgr2=0x7f13b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13b0003a00 0x7f13b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 -- 172.21.15.116:0/602547716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b00035b0 con 0x7f13b0004d90 2023-11-08T08:19:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13c50ae700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13b0005200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fc0 tx=0x7f13b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf9aba6d9047ecf server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b800ec30 con 0x7f13b0004d90 2023-11-08T08:19:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b800f040 con 0x7f13b0004d90 2023-11-08T08:19:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.208+0000 7f13bf7fe700 1 -- 172.21.15.116:0/602547716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8014480 con 0x7f13b0004d90 2023-11-08T08:19:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c6b11700 1 -- 172.21.15.116:0/602547716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 msgr2=0x7f13b0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c6b11700 1 --2- 172.21.15.116:0/602547716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13b0005200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fc0 tx=0x7f13b0005880 comp rx=0 tx=0).stop 2023-11-08T08:19:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c50ae700 1 -- 172.21.15.116:0/602547716 reap_dead start 2023-11-08T08:19:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c6b11700 1 -- 172.21.15.116:0/602547716 shutdown_connections 2023-11-08T08:19:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c6b11700 1 -- 172.21.15.116:0/602547716 >> 172.21.15.116:0/602547716 conn(0x7f13c006c010 msgr2=0x7f13c00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c6b11700 1 -- 172.21.15.116:0/602547716 shutdown_connections 2023-11-08T08:19:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.209+0000 7f13c6b11700 1 -- 172.21.15.116:0/602547716 wait complete. 2023-11-08T08:19:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.210+0000 7f13c6b11700 1 Processor -- start 2023-11-08T08:19:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.210+0000 7f13c6b11700 1 -- start start 2023-11-08T08:19:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c6b11700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13c00b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c6b11700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c015d8c0 0x7f13c00c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c6b11700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c01e6430 0x7f13c00c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c6b11700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c00c3b00 con 0x7f13b0004d90 2023-11-08T08:19:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c6b11700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015e050 con 0x7f13c01e6430 2023-11-08T08:19:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c6b11700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015e330 con 0x7f13c015d8c0 2023-11-08T08:19:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13c00b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13c00b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35154/0 (socket says 172.21.15.116:35154) 2023-11-08T08:19:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 -- 172.21.15.116:0/3392356339 learned_addr learned my addr 172.21.15.116:0/3392356339 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13bffff700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c015d8c0 0x7f13c00c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 -- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c015d8c0 msgr2=0x7f13c00c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c50ae700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c01e6430 0x7f13c00c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c015d8c0 0x7f13c00c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 -- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c01e6430 msgr2=0x7f13c00c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c01e6430 0x7f13c00c5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 -- 172.21.15.116:0/3392356339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c006d530 con 0x7f13b0004d90 2023-11-08T08:19:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13c48ad700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13c00b35a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f13b40017a0 tx=0x7f13b40076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.211+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b40077c0 con 0x7f13b0004d90 2023-11-08T08:19:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.212+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400f8d0 con 0x7f13b0004d90 2023-11-08T08:19:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.212+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400fc80 con 0x7f13b0004d90 2023-11-08T08:19:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.212+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c006d250 con 0x7f13b0004d90 2023-11-08T08:19:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.212+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c00ce740 con 0x7f13b0004d90 2023-11-08T08:19:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.213+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c00743f0 con 0x7f13b0004d90 2023-11-08T08:19:16.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.213+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13b401d460 con 0x7f13b0004d90 2023-11-08T08:19:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.213+0000 7f13bd7fa700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a40682f0 0x7f13a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.213+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f13b4085eb0 con 0x7f13b0004d90 2023-11-08T08:19:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.213+0000 7f13bffff700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a40682f0 0x7f13a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.214+0000 7f13bffff700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a40682f0 0x7f13a406a7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f13c00c65c0 tx=0x7f13ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.217+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13b40526b0 con 0x7f13b0004d90 2023-11-08T08:19:16.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.397+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f13c0071340 con 0x7f13b0004d90 2023-11-08T08:19:16.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.398+0000 7f13bd7fa700 1 -- 172.21.15.116:0/3392356339 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f13b4055cd0 con 0x7f13b0004d90 2023-11-08T08:19:16.399 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:16.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.399+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a40682f0 msgr2=0x7f13a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.399+0000 7f13c6b11700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a40682f0 0x7f13a406a7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f13c00c65c0 tx=0x7f13ac008040 comp rx=0 tx=0).stop 2023-11-08T08:19:16.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.399+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 msgr2=0x7f13c00b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.399+0000 7f13c6b11700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13c00b35a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f13b40017a0 tx=0x7f13b40076a0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 shutdown_connections 2023-11-08T08:19:16.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a40682f0 0x7f13a406a7a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c01e6430 0x7f13c00c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0004d90 0x7f13c00b35a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 --2- 172.21.15.116:0/3392356339 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13c015d8c0 0x7f13c00c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 >> 172.21.15.116:0/3392356339 conn(0x7f13c006c010 msgr2=0x7f13c015b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:16.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 shutdown_connections 2023-11-08T08:19:16.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.400+0000 7f13c6b11700 1 -- 172.21.15.116:0/3392356339 wait complete. 2023-11-08T08:19:16.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:16.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.535+0000 7f638d386700 1 Processor -- start 2023-11-08T08:19:16.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f638d386700 1 -- start start 2023-11-08T08:19:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f638d386700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f638806a980 0x7f63880ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f638d386700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6388071890 0x7f6388071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f638d386700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638815cc10 con 0x7f63880cf710 2023-11-08T08:19:16.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f638d386700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638806c9b0 con 0x7f638806a980 2023-11-08T08:19:16.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f638d386700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638806cc90 con 0x7f6388071890 2023-11-08T08:19:16.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f63867fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f638806a980 0x7f63880ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f6386ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f63880cf710 0x7f638806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54124/0 (socket says 172.21.15.116:54124) 2023-11-08T08:19:16.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.536+0000 7f6386ffd700 1 -- 172.21.15.116:0/2215108760 learned_addr learned my addr 172.21.15.116:0/2215108760 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:16.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f6385ffb700 1 -- 172.21.15.116:0/2215108760 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966103761 0 0) 0x7f638815cc10 con 0x7f63880cf710 2023-11-08T08:19:16.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f63877fe700 1 --2- 172.21.15.116:0/2215108760 >> v2:172.21.15.129:3300/0 conn(0x7f6388071890 0x7f6388071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f6385ffb700 1 -- 172.21.15.116:0/2215108760 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63700035b0 con 0x7f63880cf710 2023-11-08T08:19:16.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f63867fc700 1 -- 172.21.15.116:0/2215108760 >> v2:172.21.15.129:3300/0 conn(0x7f6388071890 msgr2=0x7f6388071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:16.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f63867fc700 1 --2- 172.21.15.116:0/2215108760 >> v2:172.21.15.129:3300/0 conn(0x7f6388071890 0x7f6388071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f63867fc700 1 -- 172.21.15.116:0/2215108760 >> v1:172.21.15.116:6789/0 conn(0x7f63880cf710 legacy=0x7f638806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f63867fc700 1 -- 172.21.15.116:0/2215108760 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638806cf70 con 0x7f638806a980 2023-11-08T08:19:16.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f63867fc700 1 --2- 172.21.15.116:0/2215108760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f638806a980 0x7f63880ce740 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f638806ad50 tx=0x7f6378009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dc80f5c7bea561 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f6385ffb700 1 -- 172.21.15.116:0/2215108760 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6378011830 con 0x7f638806a980 2023-11-08T08:19:16.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f6385ffb700 1 -- 172.21.15.116:0/2215108760 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6378011990 con 0x7f638806a980 2023-11-08T08:19:16.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.537+0000 7f6385ffb700 1 -- 172.21.15.116:0/2215108760 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6378011d20 con 0x7f638806a980 2023-11-08T08:19:16.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 -- 172.21.15.116:0/2215108760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f638806a980 msgr2=0x7f63880ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 --2- 172.21.15.116:0/2215108760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f638806a980 0x7f63880ce740 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f638806ad50 tx=0x7f6378009b20 comp rx=0 tx=0).stop 2023-11-08T08:19:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 -- 172.21.15.116:0/2215108760 shutdown_connections 2023-11-08T08:19:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 --2- 172.21.15.116:0/2215108760 >> v2:172.21.15.129:3300/0 conn(0x7f6388071890 0x7f6388071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 --2- 172.21.15.116:0/2215108760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f638806a980 0x7f63880ce740 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 -- 172.21.15.116:0/2215108760 >> 172.21.15.116:0/2215108760 conn(0x7f638806c010 msgr2=0x7f63880cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 -- 172.21.15.116:0/2215108760 shutdown_connections 2023-11-08T08:19:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.538+0000 7f638d386700 1 -- 172.21.15.116:0/2215108760 wait complete. 2023-11-08T08:19:16.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.539+0000 7f638d386700 1 Processor -- start 2023-11-08T08:19:16.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.539+0000 7f638d386700 1 -- start start 2023-11-08T08:19:16.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f638806a980 0x7f63881f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 0x7f63880b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63880cf710 0x7f63881550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638815a6c0 con 0x7f6388071890 2023-11-08T08:19:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388159b40 con 0x7f638806a980 2023-11-08T08:19:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6388159e20 con 0x7f63880cf710 2023-11-08T08:19:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 0x7f63880b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 0x7f63880b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35170/0 (socket says 172.21.15.116:35170) 2023-11-08T08:19:16.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 -- 172.21.15.116:0/1939875902 learned_addr learned my addr 172.21.15.116:0/1939875902 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:16.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f6386ffd700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f638806a980 0x7f63881f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63877fe700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63880cf710 0x7f63881550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 -- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63880cf710 msgr2=0x7f63881550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63880cf710 0x7f63881550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 -- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f638806a980 msgr2=0x7f63881f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f638806a980 0x7f63881f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 -- 172.21.15.116:0/1939875902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780094a0 con 0x7f6388071890 2023-11-08T08:19:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f6386ffd700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f638806a980 0x7f63881f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f63867fc700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 0x7f63880b3bd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6378009b60 tx=0x7f6378004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63780036a0 con 0x7f6388071890 2023-11-08T08:19:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637801f070 con 0x7f6388071890 2023-11-08T08:19:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6378003a40 con 0x7f6388071890 2023-11-08T08:19:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638806d530 con 0x7f6388071890 2023-11-08T08:19:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.540+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638806d250 con 0x7f6388071890 2023-11-08T08:19:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.541+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6388160480 con 0x7f6388071890 2023-11-08T08:19:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.543+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f637800f050 con 0x7f6388071890 2023-11-08T08:19:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.543+0000 7f636f7fe700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63740682f0 0x7f637406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.543+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6378021070 con 0x7f6388071890 2023-11-08T08:19:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.543+0000 7f6386ffd700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63740682f0 0x7f637406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.543+0000 7f6386ffd700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63740682f0 0x7f637406a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f637c0011e0 tx=0x7f637c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.547+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f63780540a0 con 0x7f6388071890 2023-11-08T08:19:16.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.728+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63880743f0 con 0x7f6388071890 2023-11-08T08:19:16.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.729+0000 7f636f7fe700 1 -- 172.21.15.116:0/1939875902 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6378085e60 con 0x7f6388071890 2023-11-08T08:19:16.730 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:16.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63740682f0 msgr2=0x7f637406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63740682f0 0x7f637406a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f637c0011e0 tx=0x7f637c009040 comp rx=0 tx=0).stop 2023-11-08T08:19:16.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 msgr2=0x7f63880b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 0x7f63880b3bd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6378009b60 tx=0x7f6378004750 comp rx=0 tx=0).stop 2023-11-08T08:19:16.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 shutdown_connections 2023-11-08T08:19:16.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f63880cf710 0x7f63881550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f63740682f0 0x7f637406a7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f638806a980 0x7f63881f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 --2- 172.21.15.116:0/1939875902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6388071890 0x7f63880b3bd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 >> 172.21.15.116:0/1939875902 conn(0x7f638806c010 msgr2=0x7f638806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 shutdown_connections 2023-11-08T08:19:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.731+0000 7f638d386700 1 -- 172.21.15.116:0/1939875902 wait complete. 2023-11-08T08:19:16.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 Processor -- start 2023-11-08T08:19:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 -- start start 2023-11-08T08:19:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca200cf710 0x7fca200ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fca20071890 0x7fca20071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2015cc10 con 0x7fca2006a270 2023-11-08T08:19:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2006c9b0 con 0x7fca200cf710 2023-11-08T08:19:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.865+0000 7fca25f58700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2006cc90 con 0x7fca20071890 2023-11-08T08:19:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1f7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca200cf710 0x7fca200ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1ffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fca2006a270 0x7fca200b1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54132/0 (socket says 172.21.15.116:54132) 2023-11-08T08:19:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1ffff700 1 -- 172.21.15.116:0/2152732297 learned_addr learned my addr 172.21.15.116:0/2152732297 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1effd700 1 --2- 172.21.15.116:0/2152732297 >> v2:172.21.15.129:3300/0 conn(0x7fca20071890 0x7fca20071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1f7fe700 1 -- 172.21.15.116:0/2152732297 >> v2:172.21.15.129:3300/0 conn(0x7fca20071890 msgr2=0x7fca20071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1f7fe700 1 --2- 172.21.15.116:0/2152732297 >> v2:172.21.15.129:3300/0 conn(0x7fca20071890 0x7fca20071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1f7fe700 1 -- 172.21.15.116:0/2152732297 >> v1:172.21.15.116:6789/0 conn(0x7fca2006a270 legacy=0x7fca200b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1f7fe700 1 -- 172.21.15.116:0/2152732297 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2006cf70 con 0x7fca200cf710 2023-11-08T08:19:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1e7fc700 1 -- 172.21.15.116:0/2152732297 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678461705 0 0) 0x7fca2015cc10 con 0x7fca2006a270 2023-11-08T08:19:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1f7fe700 1 --2- 172.21.15.116:0/2152732297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca200ce740 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca10009840 tx=0x7fca10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd4db9137e4b3d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1e7fc700 1 -- 172.21.15.116:0/2152732297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10011830 con 0x7fca200cf710 2023-11-08T08:19:16.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1e7fc700 1 -- 172.21.15.116:0/2152732297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca10011990 con 0x7fca200cf710 2023-11-08T08:19:16.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.866+0000 7fca1e7fc700 1 -- 172.21.15.116:0/2152732297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10011d20 con 0x7fca200cf710 2023-11-08T08:19:16.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 -- 172.21.15.116:0/2152732297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 msgr2=0x7fca200ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 --2- 172.21.15.116:0/2152732297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca200ce740 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca10009840 tx=0x7fca10009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:16.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 -- 172.21.15.116:0/2152732297 shutdown_connections 2023-11-08T08:19:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 --2- 172.21.15.116:0/2152732297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca200ce740 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 --2- 172.21.15.116:0/2152732297 >> v2:172.21.15.129:3300/0 conn(0x7fca20071890 0x7fca20071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 -- 172.21.15.116:0/2152732297 >> 172.21.15.116:0/2152732297 conn(0x7fca2006c010 msgr2=0x7fca200cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 -- 172.21.15.116:0/2152732297 shutdown_connections 2023-11-08T08:19:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.867+0000 7fca25f58700 1 -- 172.21.15.116:0/2152732297 wait complete. 2023-11-08T08:19:16.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.868+0000 7fca25f58700 1 Processor -- start 2023-11-08T08:19:16.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.868+0000 7fca25f58700 1 -- start start 2023-11-08T08:19:16.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fca2006a270 0x7fca200aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca20071890 0x7fca20143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca201470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca200af520 con 0x7fca200cf710 2023-11-08T08:19:16.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2015fce0 con 0x7fca20071890 2023-11-08T08:19:16.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2015d270 con 0x7fca2006a270 2023-11-08T08:19:16.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca201470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca201470b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35178/0 (socket says 172.21.15.116:35178) 2023-11-08T08:19:16.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 -- 172.21.15.116:0/3355830644 learned_addr learned my addr 172.21.15.116:0/3355830644 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:16.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1effd700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca20071890 0x7fca20143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1f7fe700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fca2006a270 0x7fca200aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 -- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fca2006a270 msgr2=0x7fca200aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fca2006a270 0x7fca200aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 -- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca20071890 msgr2=0x7fca20143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:16.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca20071890 0x7fca20143ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:16.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 -- 172.21.15.116:0/3355830644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca100094a0 con 0x7fca200cf710 2023-11-08T08:19:16.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1effd700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca20071890 0x7fca20143ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:16.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca1ffff700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca201470b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fca18002230 tx=0x7fca1800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1800ac20 con 0x7fca200cf710 2023-11-08T08:19:16.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1800ada0 con 0x7fca200cf710 2023-11-08T08:19:16.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2006d530 con 0x7fca200cf710 2023-11-08T08:19:16.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18007310 con 0x7fca200cf710 2023-11-08T08:19:16.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.869+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2006d250 con 0x7fca200cf710 2023-11-08T08:19:16.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.870+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2006a6b0 con 0x7fca200cf710 2023-11-08T08:19:16.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.871+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca1801a030 con 0x7fca200cf710 2023-11-08T08:19:16.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.871+0000 7fca17fff700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca000682f0 0x7fca0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:16.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.871+0000 7fca1f7fe700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca000682f0 0x7fca0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:16.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.871+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fca18085a50 con 0x7fca200cf710 2023-11-08T08:19:16.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.872+0000 7fca1f7fe700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca000682f0 0x7fca0006a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fca100090b0 tx=0x7fca1001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:16.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:16.875+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca180522b0 con 0x7fca200cf710 2023-11-08T08:19:17.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.072+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fca2015fe90 con 0x7fca200cf710 2023-11-08T08:19:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.072+0000 7fca17fff700 1 -- 172.21.15.116:0/3355830644 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fca2015fe90 con 0x7fca200cf710 2023-11-08T08:19:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca000682f0 msgr2=0x7fca0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca000682f0 0x7fca0006a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fca100090b0 tx=0x7fca1001c040 comp rx=0 tx=0).stop 2023-11-08T08:19:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 msgr2=0x7fca201470b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca201470b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fca18002230 tx=0x7fca1800a060 comp rx=0 tx=0).stop 2023-11-08T08:19:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 shutdown_connections 2023-11-08T08:19:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca200cf710 0x7fca201470b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fca2006a270 0x7fca200aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca000682f0 0x7fca0006a7a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 --2- 172.21.15.116:0/3355830644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca20071890 0x7fca20143ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 >> 172.21.15.116:0/3355830644 conn(0x7fca2006c010 msgr2=0x7fca200b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 shutdown_connections 2023-11-08T08:19:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.075+0000 7fca25f58700 1 -- 172.21.15.116:0/3355830644 wait complete. 2023-11-08T08:19:17.079 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:17.087 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-08T08:19:17.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:17.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:17.126 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:17.126 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:19:17.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:17.128 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-08T08:19:17.193 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:19:17.194 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:17.218 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:19:17.218 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:17.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.399+0000 7f13f0f45700 1 Processor -- start 2023-11-08T08:19:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.399+0000 7f13f0f45700 1 -- start start 2023-11-08T08:19:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13f0f45700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13ec0cff90 0x7f13ec15f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13f0f45700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0cfb00 con 0x7f13ec0b4660 2023-11-08T08:19:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13f0f45700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec15aa20 con 0x7f13ec071f70 2023-11-08T08:19:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13f0f45700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec0cf4d0 con 0x7f13ec0cff90 2023-11-08T08:19:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ead9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13ec0cff90 0x7f13ec15f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ea59c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13ec0b4660 0x7f13ec06b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54142/0 (socket says 172.21.15.116:54142) 2023-11-08T08:19:17.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ea59c700 1 -- 172.21.15.116:0/2586636509 learned_addr learned my addr 172.21.15.116:0/2586636509 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:17.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ead9d700 1 -- 172.21.15.116:0/2586636509 >> v1:172.21.15.129:6789/0 conn(0x7f13ec071f70 legacy=0x7f13ec0b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ead9d700 1 -- 172.21.15.116:0/2586636509 >> v1:172.21.15.116:6789/0 conn(0x7f13ec0b4660 legacy=0x7f13ec06b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ead9d700 1 -- 172.21.15.116:0/2586636509 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec072550 con 0x7f13ec0cff90 2023-11-08T08:19:17.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13e959a700 1 -- 172.21.15.116:0/2586636509 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417885292 0 0) 0x7f13ec0cfb00 con 0x7f13ec0b4660 2023-11-08T08:19:17.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13ead9d700 1 --2- 172.21.15.116:0/2586636509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec0cff90 0x7f13ec15f170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009d90 tx=0x7f13e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70aaf64e0db31bc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:17.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13e959a700 1 -- 172.21.15.116:0/2586636509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e400e070 con 0x7f13ec0cff90 2023-11-08T08:19:17.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13e959a700 1 -- 172.21.15.116:0/2586636509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e400bbd0 con 0x7f13ec0cff90 2023-11-08T08:19:17.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.400+0000 7f13e959a700 1 -- 172.21.15.116:0/2586636509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4012430 con 0x7f13ec0cff90 2023-11-08T08:19:17.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 -- 172.21.15.116:0/2586636509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec0cff90 msgr2=0x7f13ec15f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 --2- 172.21.15.116:0/2586636509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec0cff90 0x7f13ec15f170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009d90 tx=0x7f13e400b330 comp rx=0 tx=0).stop 2023-11-08T08:19:17.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 -- 172.21.15.116:0/2586636509 shutdown_connections 2023-11-08T08:19:17.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 --2- 172.21.15.116:0/2586636509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec0cff90 0x7f13ec15f170 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 -- 172.21.15.116:0/2586636509 >> 172.21.15.116:0/2586636509 conn(0x7f13ec0b2fc0 msgr2=0x7f13ec071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:17.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 -- 172.21.15.116:0/2586636509 shutdown_connections 2023-11-08T08:19:17.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.401+0000 7f13f0f45700 1 -- 172.21.15.116:0/2586636509 wait complete. 2023-11-08T08:19:17.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.402+0000 7f13f0f45700 1 Processor -- start 2023-11-08T08:19:17.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.402+0000 7f13f0f45700 1 -- start start 2023-11-08T08:19:17.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13f0f45700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 0x7f13ec0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13f0f45700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13ec0b4660 0x7f13ec155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13f0f45700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13ec0cff90 0x7f13ec159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13f0f45700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec0b64b0 con 0x7f13ec071f70 2023-11-08T08:19:17.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13f0f45700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec0b58a0 con 0x7f13ec0b4660 2023-11-08T08:19:17.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13f0f45700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec0b5b80 con 0x7f13ec0cff90 2023-11-08T08:19:17.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 0x7f13ec0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 0x7f13ec0c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35186/0 (socket says 172.21.15.116:35186) 2023-11-08T08:19:17.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 -- 172.21.15.116:0/3202082543 learned_addr learned my addr 172.21.15.116:0/3202082543 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:17.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ead9d700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13ec0cff90 0x7f13ec159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13e9d9b700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13ec0b4660 0x7f13ec155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 -- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13ec0cff90 msgr2=0x7f13ec159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13ec0cff90 0x7f13ec159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 -- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13ec0b4660 msgr2=0x7f13ec155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13ec0b4660 0x7f13ec155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13ea59c700 1 -- 172.21.15.116:0/3202082543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e4009a70 con 0x7f13ec071f70 2023-11-08T08:19:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.403+0000 7f13e9d9b700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13ec0b4660 0x7f13ec155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.404+0000 7f13ea59c700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 0x7f13ec0c9b80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00c520 tx=0x7f13dc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.404+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc004c80 con 0x7f13ec071f70 2023-11-08T08:19:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.404+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc004e00 con 0x7f13ec071f70 2023-11-08T08:19:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.404+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec0b5140 con 0x7f13ec071f70 2023-11-08T08:19:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.404+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc007310 con 0x7f13ec071f70 2023-11-08T08:19:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.404+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec0cf790 con 0x7f13ec071f70 2023-11-08T08:19:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.405+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec15f6b0 con 0x7f13ec071f70 2023-11-08T08:19:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.405+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13dc01a020 con 0x7f13ec071f70 2023-11-08T08:19:17.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.406+0000 7f13daffd700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13cc0682f0 0x7f13cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.406+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f13dc085fb0 con 0x7f13ec071f70 2023-11-08T08:19:17.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.406+0000 7f13e9d9b700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13cc0682f0 0x7f13cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.406+0000 7f13e9d9b700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13cc0682f0 0x7f13cc06a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f13ec158830 tx=0x7f13e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.410+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13dc0527a0 con 0x7f13ec071f70 2023-11-08T08:19:17.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.595+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13ec0743f0 con 0x7f13ec071f70 2023-11-08T08:19:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.596+0000 7f13daffd700 1 -- 172.21.15.116:0/3202082543 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f13dc055dc0 con 0x7f13ec071f70 2023-11-08T08:19:17.597 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:17.598 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13cc0682f0 msgr2=0x7f13cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13cc0682f0 0x7f13cc06a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f13ec158830 tx=0x7f13e0009040 comp rx=0 tx=0).stop 2023-11-08T08:19:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 msgr2=0x7f13ec0c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 0x7f13ec0c9b80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00c520 tx=0x7f13dc00c8e0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 shutdown_connections 2023-11-08T08:19:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13ec0cff90 0x7f13ec159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13ec071f70 0x7f13ec0c9b80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13cc0682f0 0x7f13cc06a7a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 --2- 172.21.15.116:0/3202082543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13ec0b4660 0x7f13ec155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 >> 172.21.15.116:0/3202082543 conn(0x7f13ec0b2fc0 msgr2=0x7f13ec156400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 shutdown_connections 2023-11-08T08:19:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.598+0000 7f13f0f45700 1 -- 172.21.15.116:0/3202082543 wait complete. 2023-11-08T08:19:17.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:17.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:17.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.734+0000 7f5b8c5cc700 1 Processor -- start 2023-11-08T08:19:17.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.734+0000 7f5b8c5cc700 1 -- start start 2023-11-08T08:19:17.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.735+0000 7f5b8c5cc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 0x7f5b8406a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.735+0000 7f5b8c5cc700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8406ac10 con 0x7f5b8415c810 2023-11-08T08:19:17.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.735+0000 7f5b8c5cc700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b84160a50 con 0x7f5b840718e0 2023-11-08T08:19:17.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.735+0000 7f5b8c5cc700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8415cbe0 con 0x7f5b8406a270 2023-11-08T08:19:17.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8a368700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5b8415c810 0x7f5b84160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54150/0 (socket says 172.21.15.116:54150) 2023-11-08T08:19:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8a368700 1 -- 172.21.15.116:0/3794064140 learned_addr learned my addr 172.21.15.116:0/3794064140 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 0x7f5b8406a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941187094 0 0) 0x7f5b8406ac10 con 0x7f5b8415c810 2023-11-08T08:19:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b700035b0 con 0x7f5b8415c810 2023-11-08T08:19:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626855696 0 0) 0x7f5b84160a50 con 0x7f5b840718e0 2023-11-08T08:19:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b8406ac10 con 0x7f5b840718e0 2023-11-08T08:19:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717411122 0 0) 0x7f5b700035b0 con 0x7f5b8415c810 2023-11-08T08:19:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 msgr2=0x7f5b8406a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:17.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 0x7f5b8406a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 >> v1:172.21.15.129:6789/0 conn(0x7f5b840718e0 legacy=0x7f5b8406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84160d30 con 0x7f5b8415c810 2023-11-08T08:19:17.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f5b74001670 con 0x7f5b8415c810 2023-11-08T08:19:17.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 >> v1:172.21.15.116:6789/0 conn(0x7f5b8415c810 legacy=0x7f5b84160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70003bc0 0x7f5b70003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 0x7f5b70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b70004d90 0x7f5b70005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8415cbe0 con 0x7f5b70004d90 2023-11-08T08:19:17.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7000bac0 con 0x7f5b70003bc0 2023-11-08T08:19:17.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7000bc80 con 0x7f5b8406a270 2023-11-08T08:19:17.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b70004d90 0x7f5b70005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8a368700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70003bc0 0x7f5b70003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b89b67700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 0x7f5b70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 -- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 msgr2=0x7f5b70004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:17.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b8406a270 0x7f5b70004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 -- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70003bc0 msgr2=0x7f5b70003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:17.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70003bc0 0x7f5b70003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8ab69700 1 -- 172.21.15.116:0/3794064140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b700035b0 con 0x7f5b70004d90 2023-11-08T08:19:17.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.736+0000 7f5b8a368700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70003bc0 0x7f5b70003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:17.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b70004d90 0x7f5b70005200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009f90 tx=0x7f5b70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c436153c6488d56 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:17.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c00ec30 con 0x7f5b70004d90 2023-11-08T08:19:17.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c00f040 con 0x7f5b70004d90 2023-11-08T08:19:17.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b89366700 1 -- 172.21.15.116:0/3794064140 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c014480 con 0x7f5b70004d90 2023-11-08T08:19:17.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/3794064140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b70004d90 msgr2=0x7f5b70005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/3794064140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b70004d90 0x7f5b70005200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009f90 tx=0x7f5b70005880 comp rx=0 tx=0).stop 2023-11-08T08:19:17.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8ab69700 1 -- 172.21.15.116:0/3794064140 reap_dead start 2023-11-08T08:19:17.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/3794064140 shutdown_connections 2023-11-08T08:19:17.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/3794064140 >> 172.21.15.116:0/3794064140 conn(0x7f5b8406c010 msgr2=0x7f5b840cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:17.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/3794064140 shutdown_connections 2023-11-08T08:19:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.737+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/3794064140 wait complete. 2023-11-08T08:19:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.738+0000 7f5b8c5cc700 1 Processor -- start 2023-11-08T08:19:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 -- start start 2023-11-08T08:19:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70004d90 0x7f5b840c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b84157630 0x7f5b8415f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 0x7f5b8415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8415fc90 con 0x7f5b8415f820 2023-11-08T08:19:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84158170 con 0x7f5b70004d90 2023-11-08T08:19:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8c5cc700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840c5670 con 0x7f5b84157630 2023-11-08T08:19:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8ab69700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 0x7f5b8415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8ab69700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 0x7f5b8415cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35208/0 (socket says 172.21.15.116:35208) 2023-11-08T08:19:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b8ab69700 1 -- 172.21.15.116:0/810177800 learned_addr learned my addr 172.21.15.116:0/810177800 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.739+0000 7f5b89b67700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b84157630 0x7f5b8415f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8a368700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70004d90 0x7f5b840c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8ab69700 1 -- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b84157630 msgr2=0x7f5b8415f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b84157630 0x7f5b8415f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8ab69700 1 -- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70004d90 msgr2=0x7f5b840c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70004d90 0x7f5b840c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8ab69700 1 -- 172.21.15.116:0/810177800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840ceb20 con 0x7f5b8415f820 2023-11-08T08:19:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8a368700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70004d90 0x7f5b840c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8ab69700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 0x7f5b8415cd20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b700047d0 tx=0x7f5b7c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c009410 con 0x7f5b8415f820 2023-11-08T08:19:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c00ee40 con 0x7f5b8415f820 2023-11-08T08:19:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b84161010 con 0x7f5b8415f820 2023-11-08T08:19:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c026420 con 0x7f5b8415f820 2023-11-08T08:19:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.740+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8406cf90 con 0x7f5b8415f820 2023-11-08T08:19:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.741+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b840743f0 con 0x7f5b8415f820 2023-11-08T08:19:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.742+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b7c02d070 con 0x7f5b8415f820 2023-11-08T08:19:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.742+0000 7f5b7affd700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b680682f0 0x7f5b6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.742+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5b7c087090 con 0x7f5b8415f820 2023-11-08T08:19:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.742+0000 7f5b8a368700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b680682f0 0x7f5b6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.742+0000 7f5b8a368700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b680682f0 0x7f5b6806a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5b74001010 tx=0x7f5b74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.746+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b7c0538f0 con 0x7f5b8415f820 2023-11-08T08:19:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.928+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5b840c50b0 con 0x7f5b8415f820 2023-11-08T08:19:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.928+0000 7f5b7affd700 1 -- 172.21.15.116:0/810177800 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5b7c056f10 con 0x7f5b8415f820 2023-11-08T08:19:17.930 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:17.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b680682f0 msgr2=0x7f5b6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b680682f0 0x7f5b6806a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5b74001010 tx=0x7f5b74009040 comp rx=0 tx=0).stop 2023-11-08T08:19:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 msgr2=0x7f5b8415cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 0x7f5b8415cd20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b700047d0 tx=0x7f5b7c009750 comp rx=0 tx=0).stop 2023-11-08T08:19:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 shutdown_connections 2023-11-08T08:19:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b8415f820 0x7f5b8415cd20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b680682f0 0x7f5b6806a7a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b70004d90 0x7f5b840c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 --2- 172.21.15.116:0/810177800 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b84157630 0x7f5b8415f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 >> 172.21.15.116:0/810177800 conn(0x7f5b8406c010 msgr2=0x7f5b8406d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:17.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 shutdown_connections 2023-11-08T08:19:17.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:17.930+0000 7f5b8c5cc700 1 -- 172.21.15.116:0/810177800 wait complete. 2023-11-08T08:19:17.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.067+0000 7f121fa04700 1 Processor -- start 2023-11-08T08:19:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.067+0000 7f121fa04700 1 -- start start 2023-11-08T08:19:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121fa04700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f12180ccda0 0x7f121806b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121fa04700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12180cdf00 con 0x7f1218158d50 2023-11-08T08:19:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121fa04700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12180c9110 con 0x7f1218158980 2023-11-08T08:19:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121fa04700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121806d170 con 0x7f12180ccda0 2023-11-08T08:19:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121dfa1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f12180ccda0 0x7f121806b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121cf9f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1218158d50 0x7f12180cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54154/0 (socket says 172.21.15.116:54154) 2023-11-08T08:19:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121dfa1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f12180ccda0 0x7f121806b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35222/0 (socket says 172.21.15.116:35222) 2023-11-08T08:19:18.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121cf9f700 1 -- 172.21.15.116:0/292310462 learned_addr learned my addr 172.21.15.116:0/292310462 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:18.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121dfa1700 1 -- 172.21.15.116:0/292310462 >> v1:172.21.15.129:6789/0 conn(0x7f1218158980 legacy=0x7f12180ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121dfa1700 1 -- 172.21.15.116:0/292310462 >> v1:172.21.15.116:6789/0 conn(0x7f1218158d50 legacy=0x7f12180cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121dfa1700 1 -- 172.21.15.116:0/292310462 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12180c93f0 con 0x7f12180ccda0 2023-11-08T08:19:18.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f120ffff700 1 -- 172.21.15.116:0/292310462 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936709627 0 0) 0x7f12180cdf00 con 0x7f1218158d50 2023-11-08T08:19:18.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f121dfa1700 1 --2- 172.21.15.116:0/292310462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12180ccda0 0x7f121806b770 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1210009d90 tx=0x7f121000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23b09e3fb23f00a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f120ffff700 1 -- 172.21.15.116:0/292310462 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121000e070 con 0x7f12180ccda0 2023-11-08T08:19:18.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f120ffff700 1 -- 172.21.15.116:0/292310462 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121000bbd0 con 0x7f12180ccda0 2023-11-08T08:19:18.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.068+0000 7f120ffff700 1 -- 172.21.15.116:0/292310462 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210012430 con 0x7f12180ccda0 2023-11-08T08:19:18.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 -- 172.21.15.116:0/292310462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12180ccda0 msgr2=0x7f121806b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 --2- 172.21.15.116:0/292310462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12180ccda0 0x7f121806b770 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1210009d90 tx=0x7f121000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:18.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 -- 172.21.15.116:0/292310462 shutdown_connections 2023-11-08T08:19:18.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 --2- 172.21.15.116:0/292310462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12180ccda0 0x7f121806b770 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 -- 172.21.15.116:0/292310462 >> 172.21.15.116:0/292310462 conn(0x7f12180cb540 msgr2=0x7f12180cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:18.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 -- 172.21.15.116:0/292310462 shutdown_connections 2023-11-08T08:19:18.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.069+0000 7f121fa04700 1 -- 172.21.15.116:0/292310462 wait complete. 2023-11-08T08:19:18.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.070+0000 7f121fa04700 1 Processor -- start 2023-11-08T08:19:18.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.070+0000 7f121fa04700 1 -- start start 2023-11-08T08:19:18.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121fa04700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12180ccda0 0x7f1218167c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121fa04700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 0x7f12181681b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121fa04700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1218158d50 0x7f12181f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121fa04700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12180c48e0 con 0x7f1218158980 2023-11-08T08:19:18.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121fa04700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12180c4b30 con 0x7f12180ccda0 2023-11-08T08:19:18.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121fa04700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12180c4e10 con 0x7f1218158d50 2023-11-08T08:19:18.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 0x7f12181681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 0x7f12181681b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35238/0 (socket says 172.21.15.116:35238) 2023-11-08T08:19:18.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 -- 172.21.15.116:0/1520661440 learned_addr learned my addr 172.21.15.116:0/1520661440 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:18.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 -- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1218158d50 msgr2=0x7f12181f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1218158d50 0x7f12181f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 -- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12180ccda0 msgr2=0x7f1218167c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121d7a0700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12180ccda0 0x7f1218167c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12180ccda0 0x7f1218167c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 -- 172.21.15.116:0/1520661440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210009a70 con 0x7f1218158980 2023-11-08T08:19:18.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.071+0000 7f121cf9f700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 0x7f12181681b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f120800c520 tx=0x7f120800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.072+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208004c80 con 0x7f1218158980 2023-11-08T08:19:18.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.072+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208004e20 con 0x7f1218158980 2023-11-08T08:19:18.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.072+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208007310 con 0x7f1218158980 2023-11-08T08:19:18.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.072+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12180cdb90 con 0x7f1218158980 2023-11-08T08:19:18.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.072+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12180cd8b0 con 0x7f1218158980 2023-11-08T08:19:18.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.073+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12180cd350 con 0x7f1218158980 2023-11-08T08:19:18.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.073+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f120801a020 con 0x7f1218158980 2023-11-08T08:19:18.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.073+0000 7f120dffb700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11fc0682f0 0x7f11fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.074+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1208085840 con 0x7f1218158980 2023-11-08T08:19:18.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.074+0000 7f121d7a0700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11fc0682f0 0x7f11fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.074+0000 7f121d7a0700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11fc0682f0 0x7f11fc06a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1214007fd0 tx=0x7f1214009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.078+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1208052040 con 0x7f1218158980 2023-11-08T08:19:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.257+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12180743f0 con 0x7f1218158980 2023-11-08T08:19:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.258+0000 7f120dffb700 1 -- 172.21.15.116:0/1520661440 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1208055660 con 0x7f1218158980 2023-11-08T08:19:18.259 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:18.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11fc0682f0 msgr2=0x7f11fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11fc0682f0 0x7f11fc06a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1214007fd0 tx=0x7f1214009040 comp rx=0 tx=0).stop 2023-11-08T08:19:18.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 msgr2=0x7f12181681b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 0x7f12181681b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f120800c520 tx=0x7f120800c8e0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 shutdown_connections 2023-11-08T08:19:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1218158d50 0x7f12181f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11fc0682f0 0x7f11fc06a7a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12180ccda0 0x7f1218167c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 --2- 172.21.15.116:0/1520661440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218158980 0x7f12181681b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 >> 172.21.15.116:0/1520661440 conn(0x7f12180cb540 msgr2=0x7f121814fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 shutdown_connections 2023-11-08T08:19:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.260+0000 7f121fa04700 1 -- 172.21.15.116:0/1520661440 wait complete. 2023-11-08T08:19:18.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:18.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.395+0000 7fbc43fd2700 1 Processor -- start 2023-11-08T08:19:18.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.395+0000 7fbc43fd2700 1 -- start start 2023-11-08T08:19:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc43fd2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c0b1650 0x7fbc3c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc43fd2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc3c1599b0 0x7fbc3c15d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc43fd2700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c159ec0 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc43fd2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3c15aa60 con 0x7fbc3c1599b0 2023-11-08T08:19:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc43fd2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0b1f60 con 0x7fbc3c0b1650 2023-11-08T08:19:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4156d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc3c1599b0 0x7fbc3c15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4256f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbc3c0af0a0 0x7fbc3c15a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54156/0 (socket says 172.21.15.116:54156) 2023-11-08T08:19:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4256f700 1 -- 172.21.15.116:0/2095459609 learned_addr learned my addr 172.21.15.116:0/2095459609 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4156d700 1 -- 172.21.15.116:0/2095459609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c0b1650 msgr2=0x7fbc3c0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4156d700 1 --2- 172.21.15.116:0/2095459609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c0b1650 0x7fbc3c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4156d700 1 -- 172.21.15.116:0/2095459609 >> v1:172.21.15.116:6789/0 conn(0x7fbc3c0af0a0 legacy=0x7fbc3c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4156d700 1 -- 172.21.15.116:0/2095459609 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c15d8a0 con 0x7fbc3c1599b0 2023-11-08T08:19:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc40d6c700 1 -- 172.21.15.116:0/2095459609 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079364345 0 0) 0x7fbc3c159ec0 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc4156d700 1 --2- 172.21.15.116:0/2095459609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c1599b0 0x7fbc3c15d050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009840 tx=0x7fbc2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b1e5ddce0e6aac server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.396+0000 7fbc40d6c700 1 -- 172.21.15.116:0/2095459609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c011830 con 0x7fbc3c1599b0 2023-11-08T08:19:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc40d6c700 1 -- 172.21.15.116:0/2095459609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c011990 con 0x7fbc3c1599b0 2023-11-08T08:19:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc40d6c700 1 -- 172.21.15.116:0/2095459609 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c011d20 con 0x7fbc3c1599b0 2023-11-08T08:19:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 -- 172.21.15.116:0/2095459609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c1599b0 msgr2=0x7fbc3c15d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/2095459609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c1599b0 0x7fbc3c15d050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009840 tx=0x7fbc2c009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 -- 172.21.15.116:0/2095459609 shutdown_connections 2023-11-08T08:19:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/2095459609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c0b1650 0x7fbc3c0b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/2095459609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c1599b0 0x7fbc3c15d050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 -- 172.21.15.116:0/2095459609 >> 172.21.15.116:0/2095459609 conn(0x7fbc3c06c010 msgr2=0x7fbc3c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:18.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 -- 172.21.15.116:0/2095459609 shutdown_connections 2023-11-08T08:19:18.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.397+0000 7fbc43fd2700 1 -- 172.21.15.116:0/2095459609 wait complete. 2023-11-08T08:19:18.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.398+0000 7fbc43fd2700 1 Processor -- start 2023-11-08T08:19:18.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.398+0000 7fbc43fd2700 1 -- start start 2023-11-08T08:19:18.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc43fd2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 0x7fbc3c0b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc43fd2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3c0b1650 0x7fbc3c14f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc43fd2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c15c030 0x7fbc3c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc43fd2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c15c9e0 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc43fd2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c155c30 con 0x7fbc3c0b1650 2023-11-08T08:19:18.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc43fd2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c155f10 con 0x7fbc3c15c030 2023-11-08T08:19:18.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 0x7fbc3c0b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 0x7fbc3c0b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35254/0 (socket says 172.21.15.116:35254) 2023-11-08T08:19:18.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 -- 172.21.15.116:0/1012520118 learned_addr learned my addr 172.21.15.116:0/1012520118 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:18.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc4156d700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3c0b1650 0x7fbc3c14f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc4256f700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c15c030 0x7fbc3c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 -- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c15c030 msgr2=0x7fbc3c15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c15c030 0x7fbc3c15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 -- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3c0b1650 msgr2=0x7fbc3c14f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3c0b1650 0x7fbc3c14f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.399+0000 7fbc41d6e700 1 -- 172.21.15.116:0/1012520118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0094a0 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.400+0000 7fbc41d6e700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 0x7fbc3c0b3980 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800b4d0 tx=0x7fbc3800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.400+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38011830 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.400+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc38011990 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.400+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3c15de60 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.400+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38011d40 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.400+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3c15db80 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.401+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3c15d590 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.401+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbc38021070 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.402+0000 7fbc327fc700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc200682f0 0x7fbc2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.402+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbc38086120 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.402+0000 7fbc4156d700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc200682f0 0x7fbc2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.402+0000 7fbc4156d700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc200682f0 0x7fbc2006a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009840 tx=0x7fbc2c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.406+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbc38052980 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.585+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc3c0b3ec0 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.585+0000 7fbc327fc700 1 -- 172.21.15.116:0/1012520118 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc38055fa0 con 0x7fbc3c0af0a0 2023-11-08T08:19:18.587 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc200682f0 msgr2=0x7fbc2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc200682f0 0x7fbc2006a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009840 tx=0x7fbc2c008030 comp rx=0 tx=0).stop 2023-11-08T08:19:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 msgr2=0x7fbc3c0b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 0x7fbc3c0b3980 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800b4d0 tx=0x7fbc3800b890 comp rx=0 tx=0).stop 2023-11-08T08:19:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 shutdown_connections 2023-11-08T08:19:18.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbc3c15c030 0x7fbc3c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc3c0af0a0 0x7fbc3c0b3980 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc200682f0 0x7fbc2006a7a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 --2- 172.21.15.116:0/1012520118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3c0b1650 0x7fbc3c14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 >> 172.21.15.116:0/1012520118 conn(0x7fbc3c06c010 msgr2=0x7fbc3c0cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 shutdown_connections 2023-11-08T08:19:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.587+0000 7fbc43fd2700 1 -- 172.21.15.116:0/1012520118 wait complete. 2023-11-08T08:19:18.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.722+0000 7f4addfa2700 1 Processor -- start 2023-11-08T08:19:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.722+0000 7f4addfa2700 1 -- start start 2023-11-08T08:19:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.722+0000 7f4addfa2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4ad815c810 0x7f4ad8160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.722+0000 7f4addfa2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ad806a270 0x7f4ad806a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.722+0000 7f4addfa2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad806abb0 con 0x7f4ad80718e0 2023-11-08T08:19:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4addfa2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad815cbe0 con 0x7f4ad806a270 2023-11-08T08:19:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4addfa2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad81607a0 con 0x7f4ad815c810 2023-11-08T08:19:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ad806a270 0x7f4ad806a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ad806a270 0x7f4ad806a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35258/0 (socket says 172.21.15.116:35258) 2023-11-08T08:19:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 -- 172.21.15.116:0/1760528953 learned_addr learned my addr 172.21.15.116:0/1760528953 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad77fe700 1 --2- 172.21.15.116:0/1760528953 >> v2:172.21.15.129:3300/0 conn(0x7f4ad815c810 0x7f4ad8160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 -- 172.21.15.116:0/1760528953 >> v2:172.21.15.129:3300/0 conn(0x7f4ad815c810 msgr2=0x7f4ad8160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 --2- 172.21.15.116:0/1760528953 >> v2:172.21.15.129:3300/0 conn(0x7f4ad815c810 0x7f4ad8160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 -- 172.21.15.116:0/1760528953 >> v1:172.21.15.129:6789/0 conn(0x7f4ad80718e0 legacy=0x7f4ad806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:18.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 -- 172.21.15.116:0/1760528953 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8160a50 con 0x7f4ad806a270 2023-11-08T08:19:18.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad7fff700 1 --2- 172.21.15.116:0/1760528953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad806a270 0x7f4ad806a670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80b27c0 tx=0x7f4ad0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c237f20845096605 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad67fc700 1 -- 172.21.15.116:0/1760528953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0004230 con 0x7f4ad806a270 2023-11-08T08:19:18.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad67fc700 1 -- 172.21.15.116:0/1760528953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad0004390 con 0x7f4ad806a270 2023-11-08T08:19:18.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.723+0000 7f4ad67fc700 1 -- 172.21.15.116:0/1760528953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad00036a0 con 0x7f4ad806a270 2023-11-08T08:19:18.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 -- 172.21.15.116:0/1760528953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad806a270 msgr2=0x7f4ad806a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 --2- 172.21.15.116:0/1760528953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad806a270 0x7f4ad806a670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80b27c0 tx=0x7f4ad0009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 -- 172.21.15.116:0/1760528953 shutdown_connections 2023-11-08T08:19:18.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 --2- 172.21.15.116:0/1760528953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad806a270 0x7f4ad806a670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 --2- 172.21.15.116:0/1760528953 >> v2:172.21.15.129:3300/0 conn(0x7f4ad815c810 0x7f4ad8160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 -- 172.21.15.116:0/1760528953 >> 172.21.15.116:0/1760528953 conn(0x7f4ad806c010 msgr2=0x7f4ad80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:18.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 -- 172.21.15.116:0/1760528953 shutdown_connections 2023-11-08T08:19:18.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.724+0000 7f4addfa2700 1 -- 172.21.15.116:0/1760528953 wait complete. 2023-11-08T08:19:18.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.725+0000 7f4addfa2700 1 Processor -- start 2023-11-08T08:19:18.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 -- start start 2023-11-08T08:19:18.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ad806a270 0x7f4ad8147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 0x7f4ad815d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ad815c810 0x7f4ad81578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8147660 con 0x7f4ad80718e0 2023-11-08T08:19:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad815db00 con 0x7f4ad815c810 2023-11-08T08:19:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.726+0000 7f4addfa2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8158390 con 0x7f4ad806a270 2023-11-08T08:19:18.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 0x7f4ad815d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 0x7f4ad815d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35260/0 (socket says 172.21.15.116:35260) 2023-11-08T08:19:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 -- 172.21.15.116:0/545544322 learned_addr learned my addr 172.21.15.116:0/545544322 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad7fff700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ad815c810 0x7f4ad81578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad77fe700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ad806a270 0x7f4ad8147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 -- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ad806a270 msgr2=0x7f4ad8147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ad806a270 0x7f4ad8147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 -- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ad815c810 msgr2=0x7f4ad81578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ad815c810 0x7f4ad81578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 -- 172.21.15.116:0/545544322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad0009550 con 0x7f4ad80718e0 2023-11-08T08:19:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4ad6ffd700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 0x7f4ad815d5c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c4d0 tx=0x7f4ac000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0011790 con 0x7f4ad80718e0 2023-11-08T08:19:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac00118f0 con 0x7f4ad80718e0 2023-11-08T08:19:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad8161010 con 0x7f4ad80718e0 2023-11-08T08:19:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0011ca0 con 0x7f4ad80718e0 2023-11-08T08:19:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.727+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad8160d30 con 0x7f4ad80718e0 2023-11-08T08:19:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.728+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad80ceb20 con 0x7f4ad80718e0 2023-11-08T08:19:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.728+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ac0010050 con 0x7f4ad80718e0 2023-11-08T08:19:18.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.729+0000 7f4acffff700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ab80682f0 0x7f4ab806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:18.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.729+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4ac0085a10 con 0x7f4ad80718e0 2023-11-08T08:19:18.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.729+0000 7f4ad77fe700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ab80682f0 0x7f4ab806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:18.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.729+0000 7f4ad77fe700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ab80682f0 0x7f4ab806a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009a60 tx=0x7f4ac8003a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:18.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.733+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ac0052270 con 0x7f4ad80718e0 2023-11-08T08:19:18.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:19:18.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:19:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.917+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ad80743f0 con 0x7f4ad80718e0 2023-11-08T08:19:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.917+0000 7f4acffff700 1 -- 172.21.15.116:0/545544322 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ac0055890 con 0x7f4ad80718e0 2023-11-08T08:19:18.919 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ab80682f0 msgr2=0x7f4ab806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ab80682f0 0x7f4ab806a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009a60 tx=0x7f4ac8003a20 comp rx=0 tx=0).stop 2023-11-08T08:19:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 msgr2=0x7f4ad815d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 0x7f4ad815d5c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c4d0 tx=0x7f4ac000c890 comp rx=0 tx=0).stop 2023-11-08T08:19:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 shutdown_connections 2023-11-08T08:19:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ab80682f0 0x7f4ab806a7a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ad815c810 0x7f4ad81578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ad806a270 0x7f4ad8147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 --2- 172.21.15.116:0/545544322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad80718e0 0x7f4ad815d5c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 >> 172.21.15.116:0/545544322 conn(0x7f4ad806c010 msgr2=0x7f4ad806df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 shutdown_connections 2023-11-08T08:19:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:18.919+0000 7f4addfa2700 1 -- 172.21.15.116:0/545544322 wait complete. 2023-11-08T08:19:18.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:19.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.052+0000 7f35205e2700 1 Processor -- start 2023-11-08T08:19:19.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.053+0000 7f35205e2700 1 -- start start 2023-11-08T08:19:19.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.053+0000 7f35205e2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 0x7f35180ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.053+0000 7f35205e2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 0x7f3518071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.053+0000 7f35205e2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351815cc10 con 0x7f3518071890 2023-11-08T08:19:19.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.053+0000 7f35205e2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351806c9b0 con 0x7f35180cf710 2023-11-08T08:19:19.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.053+0000 7f35205e2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351806cc90 con 0x7f351806a980 2023-11-08T08:19:19.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 0x7f3518071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351e37e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 0x7f35180ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 0x7f3518071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:53122/0 (socket says 172.21.15.116:53122) 2023-11-08T08:19:19.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351db7d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3518071890 0x7f351806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:51270/0 (socket says 172.21.15.116:51270) 2023-11-08T08:19:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351e37e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 0x7f35180ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:43206/0 (socket says 172.21.15.116:43206) 2023-11-08T08:19:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 -- 172.21.15.116:0/1834587550 learned_addr learned my addr 172.21.15.116:0/1834587550 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 -- 172.21.15.116:0/1834587550 >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 msgr2=0x7f35180ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 --2- 172.21.15.116:0/1834587550 >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 0x7f35180ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 -- 172.21.15.116:0/1834587550 >> v1:172.21.15.129:6789/0 conn(0x7f3518071890 legacy=0x7f351806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 -- 172.21.15.116:0/1834587550 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351806cf70 con 0x7f351806a980 2023-11-08T08:19:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351e37e700 1 --2- 172.21.15.116:0/1834587550 >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 0x7f35180ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.054+0000 7f351eb7f700 1 --2- 172.21.15.116:0/1834587550 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 0x7f3518071220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3510009e80 tx=0x7f351000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6886e3f5bda638ed server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f351d37c700 1 -- 172.21.15.116:0/1834587550 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000e070 con 0x7f351806a980 2023-11-08T08:19:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f351d37c700 1 -- 172.21.15.116:0/1834587550 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351000b8d0 con 0x7f351806a980 2023-11-08T08:19:19.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f351d37c700 1 -- 172.21.15.116:0/1834587550 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000bc80 con 0x7f351806a980 2023-11-08T08:19:19.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 -- 172.21.15.116:0/1834587550 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 msgr2=0x7f3518071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 --2- 172.21.15.116:0/1834587550 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 0x7f3518071220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3510009e80 tx=0x7f351000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:19.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 -- 172.21.15.116:0/1834587550 shutdown_connections 2023-11-08T08:19:19.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 --2- 172.21.15.116:0/1834587550 >> v2:172.21.15.129:3300/0 conn(0x7f35180cf710 0x7f35180ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 --2- 172.21.15.116:0/1834587550 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f351806a980 0x7f3518071220 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 -- 172.21.15.116:0/1834587550 >> 172.21.15.116:0/1834587550 conn(0x7f351806c010 msgr2=0x7f35180cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:19.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 -- 172.21.15.116:0/1834587550 shutdown_connections 2023-11-08T08:19:19.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.055+0000 7f35205e2700 1 -- 172.21.15.116:0/1834587550 wait complete. 2023-11-08T08:19:19.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.056+0000 7f35205e2700 1 Processor -- start 2023-11-08T08:19:19.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 -- start start 2023-11-08T08:19:19.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 0x7f35180b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3518071890 0x7f351814b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35180cf710 0x7f35180c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351814ba70 con 0x7f351806a980 2023-11-08T08:19:19.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180afee0 con 0x7f3518071890 2023-11-08T08:19:19.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f35205e2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35180b01c0 con 0x7f35180cf710 2023-11-08T08:19:19.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f351e37e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 0x7f35180b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f351e37e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 0x7f35180b0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35272/0 (socket says 172.21.15.116:35272) 2023-11-08T08:19:19.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.057+0000 7f351e37e700 1 -- 172.21.15.116:0/3175412770 learned_addr learned my addr 172.21.15.116:0/3175412770 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:19.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351db7d700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3518071890 0x7f351814b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351eb7f700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35180cf710 0x7f35180c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351e37e700 1 -- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35180cf710 msgr2=0x7f35180c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351e37e700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35180cf710 0x7f35180c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351e37e700 1 -- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3518071890 msgr2=0x7f351814b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351e37e700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3518071890 0x7f351814b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351e37e700 1 -- 172.21.15.116:0/3175412770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35080094a0 con 0x7f351806a980 2023-11-08T08:19:19.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351db7d700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3518071890 0x7f351814b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:19.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f351e37e700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 0x7f35180b0f80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f350800ebe0 tx=0x7f3508009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800c8b0 con 0x7f351806a980 2023-11-08T08:19:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350800ca10 con 0x7f351806a980 2023-11-08T08:19:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3510009b30 con 0x7f351806a980 2023-11-08T08:19:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800cda0 con 0x7f351806a980 2023-11-08T08:19:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.058+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351806d530 con 0x7f351806a980 2023-11-08T08:19:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.059+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351806d250 con 0x7f351806a980 2023-11-08T08:19:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.059+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3508019030 con 0x7f351806a980 2023-11-08T08:19:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.060+0000 7f350effd700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34fc0682f0 0x7f34fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.060+0000 7f351db7d700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34fc0682f0 0x7f34fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.060+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3508085c90 con 0x7f351806a980 2023-11-08T08:19:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.060+0000 7f351db7d700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34fc0682f0 0x7f34fc06a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f351815d270 tx=0x7f3514007df0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.064+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35080524f0 con 0x7f351806a980 2023-11-08T08:19:19.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.258+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f35180b0dd0 con 0x7f351806a980 2023-11-08T08:19:19.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.258+0000 7f350effd700 1 -- 172.21.15.116:0/3175412770 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3508055b10 con 0x7f351806a980 2023-11-08T08:19:19.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34fc0682f0 msgr2=0x7f34fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34fc0682f0 0x7f34fc06a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f351815d270 tx=0x7f3514007df0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 msgr2=0x7f35180b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 0x7f35180b0f80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f350800ebe0 tx=0x7f3508009e20 comp rx=0 tx=0).stop 2023-11-08T08:19:19.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 shutdown_connections 2023-11-08T08:19:19.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35180cf710 0x7f35180c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f351806a980 0x7f35180b0f80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34fc0682f0 0x7f34fc06a7a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 --2- 172.21.15.116:0/3175412770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3518071890 0x7f351814b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 >> 172.21.15.116:0/3175412770 conn(0x7f351806c010 msgr2=0x7f35180ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:19.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 shutdown_connections 2023-11-08T08:19:19.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.260+0000 7f35205e2700 1 -- 172.21.15.116:0/3175412770 wait complete. 2023-11-08T08:19:19.264 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:19.273 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:19:19.274 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:19.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.416+0000 7f59a5f33700 1 Processor -- start 2023-11-08T08:19:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.416+0000 7f59a5f33700 1 -- start start 2023-11-08T08:19:19.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f59a5f33700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a0142db0 0x7f59a013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f59a5f33700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f59a00b1770 0x7f59a00b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f59a5f33700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a015e010 con 0x7f59a00ce040 2023-11-08T08:19:19.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f59a5f33700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a00b2130 con 0x7f59a00b1770 2023-11-08T08:19:19.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f59a5f33700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a013fc90 con 0x7f59a0142db0 2023-11-08T08:19:19.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f599effd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a0142db0 0x7f59a013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f599ffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f59a00b1770 0x7f59a00b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f599effd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a0142db0 0x7f59a013ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:53138/0 (socket says 172.21.15.116:53138) 2023-11-08T08:19:19.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f599effd700 1 -- 172.21.15.116:0/2253451887 learned_addr learned my addr 172.21.15.116:0/2253451887 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:19.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f599ffff700 1 -- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a0142db0 msgr2=0x7f59a013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:19.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.417+0000 7f599ffff700 1 --2- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a0142db0 0x7f59a013ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f599ffff700 1 -- 172.21.15.116:0/2253451887 >> v1:172.21.15.129:6789/0 conn(0x7f59a00ce040 legacy=0x7f59a015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f599ffff700 1 -- 172.21.15.116:0/2253451887 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a013f6d0 con 0x7f59a00b1770 2023-11-08T08:19:19.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f599ffff700 1 --2- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a00b1770 0x7f59a00b1bf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5998009d90 tx=0x7f599800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a074ddf0af5ceacc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f599e7fc700 1 -- 172.21.15.116:0/2253451887 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599800e070 con 0x7f59a00b1770 2023-11-08T08:19:19.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f599e7fc700 1 -- 172.21.15.116:0/2253451887 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599800bb20 con 0x7f59a00b1770 2023-11-08T08:19:19.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f599e7fc700 1 -- 172.21.15.116:0/2253451887 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998012430 con 0x7f59a00b1770 2023-11-08T08:19:19.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f59a5f33700 1 -- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a00b1770 msgr2=0x7f59a00b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f59a5f33700 1 --2- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a00b1770 0x7f59a00b1bf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5998009d90 tx=0x7f599800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:19.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f59a5f33700 1 -- 172.21.15.116:0/2253451887 shutdown_connections 2023-11-08T08:19:19.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f59a5f33700 1 --2- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a00b1770 0x7f59a00b1bf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f59a5f33700 1 --2- 172.21.15.116:0/2253451887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a0142db0 0x7f59a013ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.418+0000 7f59a5f33700 1 -- 172.21.15.116:0/2253451887 >> 172.21.15.116:0/2253451887 conn(0x7f59a006c010 msgr2=0x7f59a00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.419+0000 7f59a5f33700 1 -- 172.21.15.116:0/2253451887 shutdown_connections 2023-11-08T08:19:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.419+0000 7f59a5f33700 1 -- 172.21.15.116:0/2253451887 wait complete. 2023-11-08T08:19:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.420+0000 7f59a5f33700 1 Processor -- start 2023-11-08T08:19:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.420+0000 7f59a5f33700 1 -- start start 2023-11-08T08:19:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 0x7f59a00b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a00ce040 0x7f59a0154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a0142db0 0x7f59a00cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a01f6830 con 0x7f59a00b1770 2023-11-08T08:19:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0155400 con 0x7f59a0142db0 2023-11-08T08:19:19.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a01e5a90 con 0x7f59a00ce040 2023-11-08T08:19:19.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 0x7f59a00b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 0x7f59a00b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35288/0 (socket says 172.21.15.116:35288) 2023-11-08T08:19:19.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 -- 172.21.15.116:0/3947941144 learned_addr learned my addr 172.21.15.116:0/3947941144 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:19.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599ffff700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a0142db0 0x7f59a00cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599effd700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a00ce040 0x7f59a0154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 -- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a00ce040 msgr2=0x7f59a0154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a00ce040 0x7f59a0154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 -- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a0142db0 msgr2=0x7f59a00cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a0142db0 0x7f59a00cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 -- 172.21.15.116:0/3947941144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5998009a70 con 0x7f59a00b1770 2023-11-08T08:19:19.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599ffff700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a0142db0 0x7f59a00cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:19.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f599f7fe700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 0x7f59a00b4150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f599400c4c0 tx=0x7f599400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59940052d0 con 0x7f59a00b1770 2023-11-08T08:19:19.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599400ce90 con 0x7f59a00b1770 2023-11-08T08:19:19.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994006860 con 0x7f59a00b1770 2023-11-08T08:19:19.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a015d100 con 0x7f59a00b1770 2023-11-08T08:19:19.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.421+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a013f9b0 con 0x7f59a00b1770 2023-11-08T08:19:19.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.422+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a015d3e0 con 0x7f59a00b1770 2023-11-08T08:19:19.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.423+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5994010050 con 0x7f59a00b1770 2023-11-08T08:19:19.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.423+0000 7f5987fff700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f598c0682f0 0x7f598c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.423+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5994084b60 con 0x7f59a00b1770 2023-11-08T08:19:19.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.423+0000 7f599effd700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f598c0682f0 0x7f598c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.423+0000 7f599effd700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f598c0682f0 0x7f598c06a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f59a0156da0 tx=0x7f59900092c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.427+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f599404a5e0 con 0x7f59a00b1770 2023-11-08T08:19:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.609+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59a015e010 con 0x7f59a00b1770 2023-11-08T08:19:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.611+0000 7f5987fff700 1 -- 172.21.15.116:0/3947941144 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f59a015e010 con 0x7f59a00b1770 2023-11-08T08:19:19.612 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:19.612 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f598c0682f0 msgr2=0x7f598c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f598c0682f0 0x7f598c06a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f59a0156da0 tx=0x7f59900092c0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 msgr2=0x7f59a00b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 0x7f59a00b4150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f599400c4c0 tx=0x7f599400c7d0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 shutdown_connections 2023-11-08T08:19:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f598c0682f0 0x7f598c06a7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a0142db0 0x7f59a00cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a00b1770 0x7f59a00b4150 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 --2- 172.21.15.116:0/3947941144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f59a00ce040 0x7f59a0154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 >> 172.21.15.116:0/3947941144 conn(0x7f59a006c010 msgr2=0x7f59a01401a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 shutdown_connections 2023-11-08T08:19:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.613+0000 7f59a5f33700 1 -- 172.21.15.116:0/3947941144 wait complete. 2023-11-08T08:19:19.626 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:19.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.749+0000 7f27066ac700 1 Processor -- start 2023-11-08T08:19:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.749+0000 7f27066ac700 1 -- start start 2023-11-08T08:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f27066ac700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27000b1650 0x7f27000b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f27066ac700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f27000af0a0 0x7f270015a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f27066ac700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700159ec0 con 0x7f27001599b0 2023-11-08T08:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f27066ac700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27000b1f60 con 0x7f27000b1650 2023-11-08T08:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f27066ac700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270015d8a0 con 0x7f27000af0a0 2023-11-08T08:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26fffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27000b1650 0x7f27000b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26ff7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f27001599b0 0x7f270015d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54158/0 (socket says 172.21.15.116:54158) 2023-11-08T08:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26ff7fe700 1 -- 172.21.15.116:0/1940410761 learned_addr learned my addr 172.21.15.116:0/1940410761 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f2704c49700 1 --2- 172.21.15.116:0/1940410761 >> v2:172.21.15.129:3300/0 conn(0x7f27000af0a0 0x7f270015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26fffff700 1 -- 172.21.15.116:0/1940410761 >> v2:172.21.15.129:3300/0 conn(0x7f27000af0a0 msgr2=0x7f270015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26fffff700 1 --2- 172.21.15.116:0/1940410761 >> v2:172.21.15.129:3300/0 conn(0x7f27000af0a0 0x7f270015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26fffff700 1 -- 172.21.15.116:0/1940410761 >> v1:172.21.15.116:6789/0 conn(0x7f27001599b0 legacy=0x7f270015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26fffff700 1 -- 172.21.15.116:0/1940410761 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270015db80 con 0x7f27000b1650 2023-11-08T08:19:19.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26feffd700 1 -- 172.21.15.116:0/1940410761 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600086883 0 0) 0x7f2700159ec0 con 0x7f27001599b0 2023-11-08T08:19:19.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f2704c49700 1 --2- 172.21.15.116:0/1940410761 >> v2:172.21.15.129:3300/0 conn(0x7f27000af0a0 0x7f270015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:19.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26fffff700 1 --2- 172.21.15.116:0/1940410761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f27000b1a20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f270006e010 tx=0x7f26f0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b984ef5226e730d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26feffd700 1 -- 172.21.15.116:0/1940410761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0011830 con 0x7f27000b1650 2023-11-08T08:19:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26feffd700 1 -- 172.21.15.116:0/1940410761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f0011990 con 0x7f27000b1650 2023-11-08T08:19:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.750+0000 7f26feffd700 1 -- 172.21.15.116:0/1940410761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0011d20 con 0x7f27000b1650 2023-11-08T08:19:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 -- 172.21.15.116:0/1940410761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 msgr2=0x7f27000b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 --2- 172.21.15.116:0/1940410761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f27000b1a20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f270006e010 tx=0x7f26f0009b20 comp rx=0 tx=0).stop 2023-11-08T08:19:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 -- 172.21.15.116:0/1940410761 shutdown_connections 2023-11-08T08:19:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 --2- 172.21.15.116:0/1940410761 >> v2:172.21.15.129:3300/0 conn(0x7f27000af0a0 0x7f270015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 --2- 172.21.15.116:0/1940410761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f27000b1a20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 -- 172.21.15.116:0/1940410761 >> 172.21.15.116:0/1940410761 conn(0x7f270006c010 msgr2=0x7f27000cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 -- 172.21.15.116:0/1940410761 shutdown_connections 2023-11-08T08:19:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.751+0000 7f27066ac700 1 -- 172.21.15.116:0/1940410761 wait complete. 2023-11-08T08:19:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.752+0000 7f27066ac700 1 Processor -- start 2023-11-08T08:19:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.752+0000 7f27066ac700 1 -- start start 2023-11-08T08:19:19.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f27066ac700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f27000af0a0 0x7f27001ee180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f27066ac700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f2700157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f27066ac700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2700152200 0x7f2700151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f27066ac700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000b4ab0 con 0x7f27000b1650 2023-11-08T08:19:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f27066ac700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000b3ea0 con 0x7f2700152200 2023-11-08T08:19:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f27066ac700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000b4180 con 0x7f27000af0a0 2023-11-08T08:19:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f2700157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f2700157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35308/0 (socket says 172.21.15.116:35308) 2023-11-08T08:19:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 -- 172.21.15.116:0/3174169824 learned_addr learned my addr 172.21.15.116:0/3174169824 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26fffff700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f27000af0a0 0x7f27001ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f2704c49700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2700152200 0x7f2700151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 -- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f27000af0a0 msgr2=0x7f27001ee180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f27000af0a0 0x7f27001ee180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 -- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2700152200 msgr2=0x7f2700151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2700152200 0x7f2700151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 -- 172.21.15.116:0/3174169824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00094a0 con 0x7f27000b1650 2023-11-08T08:19:19.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.753+0000 7f26ff7fe700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f2700157240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c460 tx=0x7f26f400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.754+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4004c80 con 0x7f27000b1650 2023-11-08T08:19:19.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.754+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4004e00 con 0x7f27000b1650 2023-11-08T08:19:19.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.754+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270015a890 con 0x7f27000b1650 2023-11-08T08:19:19.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.754+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4007310 con 0x7f27000b1650 2023-11-08T08:19:19.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.754+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270015de60 con 0x7f27000b1650 2023-11-08T08:19:19.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.754+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270015aa20 con 0x7f27000b1650 2023-11-08T08:19:19.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.755+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f26f401a020 con 0x7f27000b1650 2023-11-08T08:19:19.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.755+0000 7f26fcff9700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26ec0682f0 0x7f26ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:19.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.756+0000 7f26fffff700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26ec0682f0 0x7f26ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:19.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.756+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f26f4085820 con 0x7f27000b1650 2023-11-08T08:19:19.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.756+0000 7f26fffff700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26ec0682f0 0x7f26ec06a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f26f00090b0 tx=0x7f26f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:19.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.760+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f26f4052020 con 0x7f27000b1650 2023-11-08T08:19:19.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.937+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f27000743f0 con 0x7f27000b1650 2023-11-08T08:19:19.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.938+0000 7f26fcff9700 1 -- 172.21.15.116:0/3174169824 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f26f4055640 con 0x7f27000b1650 2023-11-08T08:19:19.939 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:19.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26ec0682f0 msgr2=0x7f26ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26ec0682f0 0x7f26ec06a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f26f00090b0 tx=0x7f26f0008030 comp rx=0 tx=0).stop 2023-11-08T08:19:19.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 msgr2=0x7f2700157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:19.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f2700157240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c460 tx=0x7f26f400c820 comp rx=0 tx=0).stop 2023-11-08T08:19:19.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 shutdown_connections 2023-11-08T08:19:19.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26ec0682f0 0x7f26ec06a7a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2700152200 0x7f2700151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f27000af0a0 0x7f27001ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 --2- 172.21.15.116:0/3174169824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27000b1650 0x7f2700157240 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:19.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 >> 172.21.15.116:0/3174169824 conn(0x7f270006c010 msgr2=0x7f27000d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:19.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 shutdown_connections 2023-11-08T08:19:19.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:19.940+0000 7f27066ac700 1 -- 172.21.15.116:0/3174169824 wait complete. 2023-11-08T08:19:19.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:20.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.075+0000 7f6792714700 1 Processor -- start 2023-11-08T08:19:20.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.075+0000 7f6792714700 1 -- start start 2023-11-08T08:19:20.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f6792714700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f678c0b1650 0x7f678c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f6792714700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c0af0a0 0x7f678c15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f6792714700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c159ec0 con 0x7f678c1599b0 2023-11-08T08:19:20.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f6792714700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c0b1f60 con 0x7f678c0b1650 2023-11-08T08:19:20.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f6792714700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15d8a0 con 0x7f678c0af0a0 2023-11-08T08:19:20.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f678c0b1650 0x7f678c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678b7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f678c1599b0 0x7f678c15d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54172/0 (socket says 172.21.15.116:54172) 2023-11-08T08:19:20.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678b7fe700 1 -- 172.21.15.116:0/878614537 learned_addr learned my addr 172.21.15.116:0/878614537 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678bfff700 1 -- 172.21.15.116:0/878614537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c0af0a0 msgr2=0x7f678c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678bfff700 1 --2- 172.21.15.116:0/878614537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c0af0a0 0x7f678c15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678bfff700 1 -- 172.21.15.116:0/878614537 >> v1:172.21.15.116:6789/0 conn(0x7f678c1599b0 legacy=0x7f678c15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678bfff700 1 -- 172.21.15.116:0/878614537 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c15db80 con 0x7f678c0b1650 2023-11-08T08:19:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678affd700 1 -- 172.21.15.116:0/878614537 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530269572 0 0) 0x7f678c159ec0 con 0x7f678c1599b0 2023-11-08T08:19:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678bfff700 1 --2- 172.21.15.116:0/878614537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c0b1a20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6774009840 tx=0x7f6774009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee3821639584110 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678affd700 1 -- 172.21.15.116:0/878614537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011830 con 0x7f678c0b1650 2023-11-08T08:19:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678affd700 1 -- 172.21.15.116:0/878614537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774011990 con 0x7f678c0b1650 2023-11-08T08:19:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.076+0000 7f678affd700 1 -- 172.21.15.116:0/878614537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011d20 con 0x7f678c0b1650 2023-11-08T08:19:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 -- 172.21.15.116:0/878614537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 msgr2=0x7f678c0b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 --2- 172.21.15.116:0/878614537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c0b1a20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6774009840 tx=0x7f6774009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 -- 172.21.15.116:0/878614537 shutdown_connections 2023-11-08T08:19:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 --2- 172.21.15.116:0/878614537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c0b1a20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 --2- 172.21.15.116:0/878614537 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c0af0a0 0x7f678c15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 -- 172.21.15.116:0/878614537 >> 172.21.15.116:0/878614537 conn(0x7f678c06c010 msgr2=0x7f678c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 -- 172.21.15.116:0/878614537 shutdown_connections 2023-11-08T08:19:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.077+0000 7f6792714700 1 -- 172.21.15.116:0/878614537 wait complete. 2023-11-08T08:19:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.078+0000 7f6792714700 1 Processor -- start 2023-11-08T08:19:20.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.078+0000 7f6792714700 1 -- start start 2023-11-08T08:19:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f6792714700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f678c0af0a0 0x7f678c14df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f6792714700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c15b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f6792714700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c15bee0 0x7f678c0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f6792714700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15c380 con 0x7f678c0b1650 2023-11-08T08:19:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f6792714700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0cb250 con 0x7f678c0af0a0 2023-11-08T08:19:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f6792714700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0cb530 con 0x7f678c15bee0 2023-11-08T08:19:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c15b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c15b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35322/0 (socket says 172.21.15.116:35322) 2023-11-08T08:19:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 -- 172.21.15.116:0/3777577758 learned_addr learned my addr 172.21.15.116:0/3777577758 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 -- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c15bee0 msgr2=0x7f678c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c15bee0 0x7f678c0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678bfff700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f678c0af0a0 0x7f678c14df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 -- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f678c0af0a0 msgr2=0x7f678c14df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f678c0af0a0 0x7f678c14df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.079+0000 7f678b7fe700 1 -- 172.21.15.116:0/3777577758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740094a0 con 0x7f678c0b1650 2023-11-08T08:19:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.080+0000 7f678b7fe700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c15b9a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f677c00c460 tx=0x7f677c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.080+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c004c80 con 0x7f678c0b1650 2023-11-08T08:19:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.080+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c004e00 con 0x7f678c0b1650 2023-11-08T08:19:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.080+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c15a890 con 0x7f678c0b1650 2023-11-08T08:19:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.080+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c007310 con 0x7f678c0b1650 2023-11-08T08:19:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.080+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c15de60 con 0x7f678c0b1650 2023-11-08T08:19:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.081+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c15aa20 con 0x7f678c0b1650 2023-11-08T08:19:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.081+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f677c01a020 con 0x7f678c0b1650 2023-11-08T08:19:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.081+0000 7f6788ff9700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f676c0682f0 0x7f676c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.081+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f677c085810 con 0x7f678c0b1650 2023-11-08T08:19:20.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.082+0000 7f678bfff700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f676c0682f0 0x7f676c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.082+0000 7f678bfff700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f676c0682f0 0x7f676c06a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f67740090b0 tx=0x7f6774008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.086+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f677c052010 con 0x7f678c0b1650 2023-11-08T08:19:20.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.265+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f678c0743f0 con 0x7f678c0b1650 2023-11-08T08:19:20.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.266+0000 7f6788ff9700 1 -- 172.21.15.116:0/3777577758 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f677c055630 con 0x7f678c0b1650 2023-11-08T08:19:20.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f676c0682f0 msgr2=0x7f676c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f676c0682f0 0x7f676c06a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f67740090b0 tx=0x7f6774008030 comp rx=0 tx=0).stop 2023-11-08T08:19:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 msgr2=0x7f678c15b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c15b9a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f677c00c460 tx=0x7f677c00c820 comp rx=0 tx=0).stop 2023-11-08T08:19:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 shutdown_connections 2023-11-08T08:19:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f678c15bee0 0x7f678c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f676c0682f0 0x7f676c06a7a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f678c0af0a0 0x7f678c14df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 --2- 172.21.15.116:0/3777577758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f678c0b1650 0x7f678c15b9a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 >> 172.21.15.116:0/3777577758 conn(0x7f678c06c010 msgr2=0x7f678c071750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 shutdown_connections 2023-11-08T08:19:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.268+0000 7f6792714700 1 -- 172.21.15.116:0/3777577758 wait complete. 2023-11-08T08:19:20.281 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:20.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.404+0000 7f826a889700 1 Processor -- start 2023-11-08T08:19:20.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.404+0000 7f826a889700 1 -- start start 2023-11-08T08:19:20.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f826a889700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f82640cf710 0x7f82640ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f826a889700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8264071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f826a889700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826415cc10 con 0x7f826406a270 2023-11-08T08:19:20.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f826a889700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826406c9b0 con 0x7f82640cf710 2023-11-08T08:19:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f826a889700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826406cc90 con 0x7f8264071890 2023-11-08T08:19:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8268e26700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f826406a270 0x7f82640b1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54184/0 (socket says 172.21.15.116:54184) 2023-11-08T08:19:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8268e26700 1 -- 172.21.15.116:0/938377851 learned_addr learned my addr 172.21.15.116:0/938377851 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8263fff700 1 --2- 172.21.15.116:0/938377851 >> v2:172.21.15.129:3300/0 conn(0x7f82640cf710 0x7f82640ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f82637fe700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8264071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548071875 0 0) 0x7f826415cc10 con 0x7f826406a270 2023-11-08T08:19:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82500035b0 con 0x7f826406a270 2023-11-08T08:19:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3917991735 0 0) 0x7f82500035b0 con 0x7f826406a270 2023-11-08T08:19:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 msgr2=0x7f8264071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8264071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 >> v2:172.21.15.129:3300/0 conn(0x7f82640cf710 msgr2=0x7f82640ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 --2- 172.21.15.116:0/938377851 >> v2:172.21.15.129:3300/0 conn(0x7f82640cf710 0x7f82640ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826406cf70 con 0x7f826406a270 2023-11-08T08:19:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f825c0019f0 con 0x7f826406a270 2023-11-08T08:19:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 >> v1:172.21.15.116:6789/0 conn(0x7f826406a270 legacy=0x7f82640b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8250003a00 0x7f8250003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8250004430 0x7f82500048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8250005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826406c9b0 con 0x7f8250004430 2023-11-08T08:19:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826406cc90 con 0x7f8250003a00 2023-11-08T08:19:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.405+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825000ba10 con 0x7f8264071890 2023-11-08T08:19:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8250004430 0x7f82500048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f8268e26700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8250005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f8263fff700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8250003a00 0x7f8250003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 -- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 msgr2=0x7f8250005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8250005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 -- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8250003a00 msgr2=0x7f8250003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8250003a00 0x7f8250003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 -- 172.21.15.116:0/938377851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500035b0 con 0x7f8250004430 2023-11-08T08:19:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f8268e26700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8264071890 0x7f8250005060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f82637fe700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8250004430 0x7f82500048f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8254009b90 tx=0x7f8250005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d66613ee388998 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400e6c0 con 0x7f8250004430 2023-11-08T08:19:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825400e820 con 0x7f8250004430 2023-11-08T08:19:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.406+0000 7f8262ffd700 1 -- 172.21.15.116:0/938377851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400eb90 con 0x7f8250004430 2023-11-08T08:19:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f826a889700 1 -- 172.21.15.116:0/938377851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8250004430 msgr2=0x7f82500048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f826a889700 1 --2- 172.21.15.116:0/938377851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8250004430 0x7f82500048f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8254009b90 tx=0x7f8250005880 comp rx=0 tx=0).stop 2023-11-08T08:19:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f8268e26700 1 -- 172.21.15.116:0/938377851 reap_dead start 2023-11-08T08:19:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f826a889700 1 -- 172.21.15.116:0/938377851 shutdown_connections 2023-11-08T08:19:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f826a889700 1 -- 172.21.15.116:0/938377851 >> 172.21.15.116:0/938377851 conn(0x7f826406c010 msgr2=0x7f82640cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f826a889700 1 -- 172.21.15.116:0/938377851 shutdown_connections 2023-11-08T08:19:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.407+0000 7f826a889700 1 -- 172.21.15.116:0/938377851 wait complete. 2023-11-08T08:19:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.408+0000 7f826a889700 1 Processor -- start 2023-11-08T08:19:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.408+0000 7f826a889700 1 -- start start 2023-11-08T08:19:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f826a889700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8250004430 0x7f8264152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f826a889700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 0x7f8264148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f826a889700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82641f2620 0x7f82640b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f826a889700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8264149410 con 0x7f8264155620 2023-11-08T08:19:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f826a889700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826415e6d0 con 0x7f82641f2620 2023-11-08T08:19:20.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f826a889700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826415e970 con 0x7f8250004430 2023-11-08T08:19:20.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f82637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 0x7f8264148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f82637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 0x7f8264148ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35336/0 (socket says 172.21.15.116:35336) 2023-11-08T08:19:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.409+0000 7f82637fe700 1 -- 172.21.15.116:0/1481618902 learned_addr learned my addr 172.21.15.116:0/1481618902 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f8263fff700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8250004430 0x7f8264152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f8268e26700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82641f2620 0x7f82640b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f82637fe700 1 -- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8250004430 msgr2=0x7f8264152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f82637fe700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8250004430 0x7f8264152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f82637fe700 1 -- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82641f2620 msgr2=0x7f82640b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f82637fe700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82641f2620 0x7f82640b0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f82637fe700 1 -- 172.21.15.116:0/1481618902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826406d530 con 0x7f8264155620 2023-11-08T08:19:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f8268e26700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82641f2620 0x7f82640b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f82637fe700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 0x7f8264148ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f82500057a0 tx=0x7f82540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400e6c0 con 0x7f8264155620 2023-11-08T08:19:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254003ab0 con 0x7f8264155620 2023-11-08T08:19:20.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826406d250 con 0x7f8264155620 2023-11-08T08:19:20.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254003e60 con 0x7f8264155620 2023-11-08T08:19:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.410+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8264071220 con 0x7f8264155620 2023-11-08T08:19:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.411+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82640743f0 con 0x7f8264155620 2023-11-08T08:19:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.411+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8254025070 con 0x7f8264155620 2023-11-08T08:19:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.412+0000 7f8260ff9700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8244068320 0x7f824406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.412+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f82540874b0 con 0x7f8264155620 2023-11-08T08:19:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.412+0000 7f8263fff700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8244068320 0x7f824406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.412+0000 7f8263fff700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8244068320 0x7f824406a7d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f824c009c20 tx=0x7f824c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.416+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8254053d10 con 0x7f8264155620 2023-11-08T08:19:20.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.595+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82640adbb0 con 0x7f8264155620 2023-11-08T08:19:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.595+0000 7f8260ff9700 1 -- 172.21.15.116:0/1481618902 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8254057330 con 0x7f8264155620 2023-11-08T08:19:20.597 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:20.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8244068320 msgr2=0x7f824406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8244068320 0x7f824406a7d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f824c009c20 tx=0x7f824c008040 comp rx=0 tx=0).stop 2023-11-08T08:19:20.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 msgr2=0x7f8264148ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 0x7f8264148ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f82500057a0 tx=0x7f82540047c0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 shutdown_connections 2023-11-08T08:19:20.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8244068320 0x7f824406a7d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82641f2620 0x7f82640b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8250004430 0x7f8264152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 --2- 172.21.15.116:0/1481618902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8264155620 0x7f8264148ed0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 >> 172.21.15.116:0/1481618902 conn(0x7f826406c010 msgr2=0x7f82640cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:20.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 shutdown_connections 2023-11-08T08:19:20.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.597+0000 7f826a889700 1 -- 172.21.15.116:0/1481618902 wait complete. 2023-11-08T08:19:20.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.733+0000 7f3558df2700 1 Processor -- start 2023-11-08T08:19:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.733+0000 7f3558df2700 1 -- start start 2023-11-08T08:19:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3558df2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35540718e0 0x7f355406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3558df2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f355406cdc0 0x7f355406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3558df2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3554160a50 con 0x7f355415c810 2023-11-08T08:19:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3558df2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355415cbe0 con 0x7f355406cdc0 2023-11-08T08:19:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3558df2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3554160d30 con 0x7f35540718e0 2023-11-08T08:19:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3552d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f355406cdc0 0x7f355406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3552d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f355406cdc0 0x7f355406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35344/0 (socket says 172.21.15.116:35344) 2023-11-08T08:19:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3552d9d700 1 -- 172.21.15.116:0/687611881 learned_addr learned my addr 172.21.15.116:0/687611881 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3551d9b700 1 --2- 172.21.15.116:0/687611881 >> v2:172.21.15.129:3300/0 conn(0x7f35540718e0 0x7f355406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3552d9d700 1 -- 172.21.15.116:0/687611881 >> v2:172.21.15.129:3300/0 conn(0x7f35540718e0 msgr2=0x7f355406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3552d9d700 1 --2- 172.21.15.116:0/687611881 >> v2:172.21.15.129:3300/0 conn(0x7f35540718e0 0x7f355406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.734+0000 7f3552d9d700 1 -- 172.21.15.116:0/687611881 >> v1:172.21.15.129:6789/0 conn(0x7f355415c810 legacy=0x7f3554160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3552d9d700 1 -- 172.21.15.116:0/687611881 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554161010 con 0x7f355406cdc0 2023-11-08T08:19:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3552d9d700 1 --2- 172.21.15.116:0/687611881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355406cdc0 0x7f355406a270 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f354c009e50 tx=0x7f354c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da38649beff1ca2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f355159a700 1 -- 172.21.15.116:0/687611881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c00e070 con 0x7f355406cdc0 2023-11-08T08:19:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f355159a700 1 -- 172.21.15.116:0/687611881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c003a20 con 0x7f355406cdc0 2023-11-08T08:19:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f355159a700 1 -- 172.21.15.116:0/687611881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c003dd0 con 0x7f355406cdc0 2023-11-08T08:19:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 -- 172.21.15.116:0/687611881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355406cdc0 msgr2=0x7f355406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 --2- 172.21.15.116:0/687611881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355406cdc0 0x7f355406a270 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f354c009e50 tx=0x7f354c00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 -- 172.21.15.116:0/687611881 shutdown_connections 2023-11-08T08:19:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 --2- 172.21.15.116:0/687611881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355406cdc0 0x7f355406a270 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 --2- 172.21.15.116:0/687611881 >> v2:172.21.15.129:3300/0 conn(0x7f35540718e0 0x7f355406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 -- 172.21.15.116:0/687611881 >> 172.21.15.116:0/687611881 conn(0x7f355406c010 msgr2=0x7f35540cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:20.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 -- 172.21.15.116:0/687611881 shutdown_connections 2023-11-08T08:19:20.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.735+0000 7f3558df2700 1 -- 172.21.15.116:0/687611881 wait complete. 2023-11-08T08:19:20.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.737+0000 7f3558df2700 1 Processor -- start 2023-11-08T08:19:20.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.737+0000 7f3558df2700 1 -- start start 2023-11-08T08:19:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.737+0000 7f3558df2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f355406cdc0 0x7f35541f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.737+0000 7f3558df2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 0x7f35541ef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3558df2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f355415c810 0x7f3554157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3558df2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3554158260 con 0x7f35540718e0 2023-11-08T08:19:20.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3558df2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35541f2710 con 0x7f355415c810 2023-11-08T08:19:20.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3558df2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35541f29f0 con 0x7f355406cdc0 2023-11-08T08:19:20.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 0x7f35541ef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 0x7f35541ef7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35350/0 (socket says 172.21.15.116:35350) 2023-11-08T08:19:20.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 -- 172.21.15.116:0/2860629953 learned_addr learned my addr 172.21.15.116:0/2860629953 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:20.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3552d9d700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f355415c810 0x7f3554157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f355259c700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f355406cdc0 0x7f35541f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 -- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f355406cdc0 msgr2=0x7f35541f2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f355406cdc0 0x7f35541f2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 -- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f355415c810 msgr2=0x7f3554157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f355415c810 0x7f3554157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 -- 172.21.15.116:0/2860629953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c009b30 con 0x7f35540718e0 2023-11-08T08:19:20.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3552d9d700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f355415c810 0x7f3554157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:20.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3551d9b700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 0x7f35541ef7b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f354400e810 tx=0x7f354400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544013610 con 0x7f35540718e0 2023-11-08T08:19:20.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3544013770 con 0x7f35540718e0 2023-11-08T08:19:20.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35540ce740 con 0x7f35540718e0 2023-11-08T08:19:20.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544013b20 con 0x7f35540718e0 2023-11-08T08:19:20.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.738+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35540ceb20 con 0x7f35540718e0 2023-11-08T08:19:20.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.739+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35540ce8d0 con 0x7f35540718e0 2023-11-08T08:19:20.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.740+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3544023070 con 0x7f35540718e0 2023-11-08T08:19:20.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.740+0000 7f3542ffd700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35340682f0 0x7f353406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:20.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.740+0000 7f355259c700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35340682f0 0x7f353406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:20.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.740+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f35440859a0 con 0x7f35540718e0 2023-11-08T08:19:20.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.741+0000 7f355259c700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35340682f0 0x7f353406a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3548009870 tx=0x7f3548008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:20.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.744+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3544052180 con 0x7f35540718e0 2023-11-08T08:19:20.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.923+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35540743f0 con 0x7f35540718e0 2023-11-08T08:19:20.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.924+0000 7f3542ffd700 1 -- 172.21.15.116:0/2860629953 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f35440557a0 con 0x7f35540718e0 2023-11-08T08:19:20.926 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.925+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35340682f0 msgr2=0x7f353406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.925+0000 7f3558df2700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35340682f0 0x7f353406a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3548009870 tx=0x7f3548008040 comp rx=0 tx=0).stop 2023-11-08T08:19:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.925+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 msgr2=0x7f35541ef7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.925+0000 7f3558df2700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 0x7f35541ef7b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f354400e810 tx=0x7f354400ebd0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 shutdown_connections 2023-11-08T08:19:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35340682f0 0x7f353406a7a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f355415c810 0x7f3554157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f355406cdc0 0x7f35541f2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 --2- 172.21.15.116:0/2860629953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35540718e0 0x7f35541ef7b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:20.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 >> 172.21.15.116:0/2860629953 conn(0x7f355406c010 msgr2=0x7f35540b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:20.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 shutdown_connections 2023-11-08T08:19:20.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:20.926+0000 7f3558df2700 1 -- 172.21.15.116:0/2860629953 wait complete. 2023-11-08T08:19:20.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:21.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.059+0000 7f559eee1700 1 Processor -- start 2023-11-08T08:19:21.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.059+0000 7f559eee1700 1 -- start start 2023-11-08T08:19:21.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.060+0000 7f559eee1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f559806a980 0x7f55980ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.060+0000 7f559eee1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559815cc10 con 0x7f55980cf710 2023-11-08T08:19:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.060+0000 7f559eee1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559806c9b0 con 0x7f5598071890 2023-11-08T08:19:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.060+0000 7f559eee1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559806cc90 con 0x7f559806a980 2023-11-08T08:19:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.060+0000 7f559cc7d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f55980cf710 0x7f559806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54190/0 (socket says 172.21.15.116:54190) 2023-11-08T08:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.060+0000 7f559cc7d700 1 -- 172.21.15.116:0/1466641331 learned_addr learned my addr 172.21.15.116:0/1466641331 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f5597fff700 1 --2- 172.21.15.116:0/1466641331 >> v2:172.21.15.129:3300/0 conn(0x7f559806a980 0x7f55980ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042905403 0 0) 0x7f559815cc10 con 0x7f55980cf710 2023-11-08T08:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55880035b0 con 0x7f55980cf710 2023-11-08T08:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2913480561 0 0) 0x7f55880035b0 con 0x7f55980cf710 2023-11-08T08:19:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 >> v2:172.21.15.129:3300/0 conn(0x7f559806a980 msgr2=0x7f55980ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 --2- 172.21.15.116:0/1466641331 >> v2:172.21.15.129:3300/0 conn(0x7f559806a980 0x7f55980ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 >> v1:172.21.15.129:6789/0 conn(0x7f5598071890 legacy=0x7f5598160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559806cf70 con 0x7f55980cf710 2023-11-08T08:19:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f558c001670 con 0x7f55980cf710 2023-11-08T08:19:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 >> v1:172.21.15.116:6789/0 conn(0x7f55980cf710 legacy=0x7f559806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f5588003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5588004850 0x7f5588004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5588005200 0x7f558800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559806cc90 con 0x7f5588003a00 2023-11-08T08:19:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558800c110 con 0x7f5588005200 2023-11-08T08:19:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558800c2d0 con 0x7f5588004850 2023-11-08T08:19:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f5588003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f5597fff700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5588004850 0x7f5588004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559d47e700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5588005200 0x7f558800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 -- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5588004850 msgr2=0x7f5588004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:21.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5588004850 0x7f5588004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 -- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5588005200 msgr2=0x7f558800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:21.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5588005200 0x7f558800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559d47e700 1 -- 172.21.15.116:0/1466641331 reap_dead start 2023-11-08T08:19:21.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 -- 172.21.15.116:0/1466641331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880035b0 con 0x7f5588003a00 2023-11-08T08:19:21.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.061+0000 7f559cc7d700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f5588003e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f558c00c240 tx=0x7f5588005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7125e5021925f4e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c007c10 con 0x7f5588003a00 2023-11-08T08:19:21.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c00ce00 con 0x7f5588003a00 2023-11-08T08:19:21.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f55977fe700 1 -- 172.21.15.116:0/1466641331 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c005ba0 con 0x7f5588003a00 2023-11-08T08:19:21.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 -- 172.21.15.116:0/1466641331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 msgr2=0x7f5588003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f5588003e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f558c00c240 tx=0x7f5588005880 comp rx=0 tx=0).stop 2023-11-08T08:19:21.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 -- 172.21.15.116:0/1466641331 shutdown_connections 2023-11-08T08:19:21.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 --2- 172.21.15.116:0/1466641331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f5588003e10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 -- 172.21.15.116:0/1466641331 >> 172.21.15.116:0/1466641331 conn(0x7f559806c010 msgr2=0x7f55980cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:21.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 -- 172.21.15.116:0/1466641331 shutdown_connections 2023-11-08T08:19:21.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.062+0000 7f559eee1700 1 -- 172.21.15.116:0/1466641331 wait complete. 2023-11-08T08:19:21.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.063+0000 7f559eee1700 1 Processor -- start 2023-11-08T08:19:21.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.063+0000 7f559eee1700 1 -- start start 2023-11-08T08:19:21.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559eee1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f559814e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559eee1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55980b50b0 0x7f55980cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559eee1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55980b3a50 0x7f55980c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559eee1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55980b4c20 con 0x7f5588003a00 2023-11-08T08:19:21.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559eee1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55980b4010 con 0x7f55980b50b0 2023-11-08T08:19:21.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559eee1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55980b42f0 con 0x7f55980b3a50 2023-11-08T08:19:21.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559cc7d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f559814e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559cc7d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f559814e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35362/0 (socket says 172.21.15.116:35362) 2023-11-08T08:19:21.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.064+0000 7f559cc7d700 1 -- 172.21.15.116:0/2454893428 learned_addr learned my addr 172.21.15.116:0/2454893428 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:21.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559cc7d700 1 -- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55980b3a50 msgr2=0x7f55980c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559cc7d700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55980b3a50 0x7f55980c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559cc7d700 1 -- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55980b50b0 msgr2=0x7f55980cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f5597fff700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55980b50b0 0x7f55980cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559cc7d700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55980b50b0 0x7f55980cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559cc7d700 1 -- 172.21.15.116:0/2454893428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559806d530 con 0x7f5588003a00 2023-11-08T08:19:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559cc7d700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f559814e150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f558c001930 tx=0x7f558c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c004c80 con 0x7f5588003a00 2023-11-08T08:19:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c004e00 con 0x7f5588003a00 2023-11-08T08:19:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c007310 con 0x7f5588003a00 2023-11-08T08:19:21.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559806d250 con 0x7f5588003a00 2023-11-08T08:19:21.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.065+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55980ce740 con 0x7f5588003a00 2023-11-08T08:19:21.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.066+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55980743f0 con 0x7f5588003a00 2023-11-08T08:19:21.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.066+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f558c010070 con 0x7f5588003a00 2023-11-08T08:19:21.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.067+0000 7f55957fa700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f557c0682f0 0x7f557c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.067+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f558c086590 con 0x7f5588003a00 2023-11-08T08:19:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.067+0000 7f5597fff700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f557c0682f0 0x7f557c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.067+0000 7f5597fff700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f557c0682f0 0x7f557c06a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f55980ae260 tx=0x7f5584008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.071+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f558c052d70 con 0x7f5588003a00 2023-11-08T08:19:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.264+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f55980b4f00 con 0x7f5588003a00 2023-11-08T08:19:21.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.265+0000 7f55957fa700 1 -- 172.21.15.116:0/2454893428 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f558c056390 con 0x7f5588003a00 2023-11-08T08:19:21.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f557c0682f0 msgr2=0x7f557c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f557c0682f0 0x7f557c06a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f55980ae260 tx=0x7f5584008040 comp rx=0 tx=0).stop 2023-11-08T08:19:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 msgr2=0x7f559814e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f559814e150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f558c001930 tx=0x7f558c001a00 comp rx=0 tx=0).stop 2023-11-08T08:19:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 shutdown_connections 2023-11-08T08:19:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f55980b3a50 0x7f55980c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5588003a00 0x7f559814e150 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f557c0682f0 0x7f557c06a7a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 --2- 172.21.15.116:0/2454893428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55980b50b0 0x7f55980cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 >> 172.21.15.116:0/2454893428 conn(0x7f559806c010 msgr2=0x7f55980afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 shutdown_connections 2023-11-08T08:19:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.267+0000 7f559eee1700 1 -- 172.21.15.116:0/2454893428 wait complete. 2023-11-08T08:19:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:21.280 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi129.front.sepia.ceph.com... 2023-11-08T08:19:21.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:21.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-11-08T08:19:21.325 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:21.326 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:19:21.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:21.327 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-08T08:19:21.354 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:21.355 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:21.373 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:21.374 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:21.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.592+0000 7f6882dfb700 1 Processor -- start 2023-11-08T08:19:21.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.592+0000 7f6882dfb700 1 -- start start 2023-11-08T08:19:21.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6882dfb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f687c163430 0x7f687c165810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6882dfb700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c151a50 con 0x7f687c070d90 2023-11-08T08:19:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6882dfb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c15b440 con 0x7f687c06a270 2023-11-08T08:19:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6882dfb700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687c15f0c0 con 0x7f687c163430 2023-11-08T08:19:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6881398700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f687c163430 0x7f687c165810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f687bfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f687c070d90 0x7f687c0711e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54192/0 (socket says 172.21.15.116:54192) 2023-11-08T08:19:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f687bfff700 1 -- 172.21.15.116:0/2914499264 learned_addr learned my addr 172.21.15.116:0/2914499264 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6881398700 1 -- 172.21.15.116:0/2914499264 >> v1:172.21.15.129:6789/0 conn(0x7f687c06a270 legacy=0x7f687c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6881398700 1 -- 172.21.15.116:0/2914499264 >> v1:172.21.15.116:6789/0 conn(0x7f687c070d90 legacy=0x7f687c0711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6881398700 1 -- 172.21.15.116:0/2914499264 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c071ac0 con 0x7f687c163430 2023-11-08T08:19:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f687b7fe700 1 -- 172.21.15.116:0/2914499264 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822696826 0 0) 0x7f687c151a50 con 0x7f687c070d90 2023-11-08T08:19:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.593+0000 7f6881398700 1 --2- 172.21.15.116:0/2914499264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c163430 0x7f687c165810 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6874009d90 tx=0x7f687400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7fdfef2a90c94b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f687b7fe700 1 -- 172.21.15.116:0/2914499264 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400e070 con 0x7f687c163430 2023-11-08T08:19:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f687b7fe700 1 -- 172.21.15.116:0/2914499264 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400bb90 con 0x7f687c163430 2023-11-08T08:19:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f687b7fe700 1 -- 172.21.15.116:0/2914499264 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6874003a20 con 0x7f687c163430 2023-11-08T08:19:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 -- 172.21.15.116:0/2914499264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c163430 msgr2=0x7f687c165810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2914499264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c163430 0x7f687c165810 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6874009d90 tx=0x7f687400b330 comp rx=0 tx=0).stop 2023-11-08T08:19:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 -- 172.21.15.116:0/2914499264 shutdown_connections 2023-11-08T08:19:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2914499264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c163430 0x7f687c165810 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 -- 172.21.15.116:0/2914499264 >> 172.21.15.116:0/2914499264 conn(0x7f687c06c010 msgr2=0x7f687c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:21.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 -- 172.21.15.116:0/2914499264 shutdown_connections 2023-11-08T08:19:21.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.594+0000 7f6882dfb700 1 -- 172.21.15.116:0/2914499264 wait complete. 2023-11-08T08:19:21.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.595+0000 7f6882dfb700 1 Processor -- start 2023-11-08T08:19:21.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.595+0000 7f6882dfb700 1 -- start start 2023-11-08T08:19:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f6882dfb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f687c06a270 0x7f687c0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f6882dfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 0x7f687c150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f6882dfb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f687c163430 0x7f687c153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f6882dfb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0cf710 con 0x7f687c070d90 2023-11-08T08:19:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f6882dfb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0c81c0 con 0x7f687c163430 2023-11-08T08:19:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f6882dfb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0cee70 con 0x7f687c06a270 2023-11-08T08:19:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f687bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 0x7f687c150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f687bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 0x7f687c150300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35382/0 (socket says 172.21.15.116:35382) 2023-11-08T08:19:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.596+0000 7f687bfff700 1 -- 172.21.15.116:0/2410915372 learned_addr learned my addr 172.21.15.116:0/2410915372 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f687bfff700 1 -- 172.21.15.116:0/2410915372 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f687c06a270 msgr2=0x7f687c0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f687bfff700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f687c06a270 0x7f687c0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f687bfff700 1 -- 172.21.15.116:0/2410915372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f687c163430 msgr2=0x7f687c153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f687bfff700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f687c163430 0x7f687c153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f687bfff700 1 -- 172.21.15.116:0/2410915372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874009a70 con 0x7f687c070d90 2023-11-08T08:19:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f687bfff700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 0x7f687c150300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f686800c750 tx=0x7f686800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868004c80 con 0x7f687c070d90 2023-11-08T08:19:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6868004e00 con 0x7f687c070d90 2023-11-08T08:19:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c15b0d0 con 0x7f687c070d90 2023-11-08T08:19:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868007310 con 0x7f687c070d90 2023-11-08T08:19:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.597+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c15adf0 con 0x7f687c070d90 2023-11-08T08:19:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.598+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c15aac0 con 0x7f687c070d90 2023-11-08T08:19:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.598+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f686801a020 con 0x7f687c070d90 2023-11-08T08:19:21.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.599+0000 7f68797fa700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68600682f0 0x7f686006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.599+0000 7f6880b97700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68600682f0 0x7f686006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.599+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6868085920 con 0x7f687c070d90 2023-11-08T08:19:21.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.599+0000 7f6880b97700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68600682f0 0x7f686006a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6870001600 tx=0x7f6870007a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.603+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6868052120 con 0x7f687c070d90 2023-11-08T08:19:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.785+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f687c0743f0 con 0x7f687c070d90 2023-11-08T08:19:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.786+0000 7f68797fa700 1 -- 172.21.15.116:0/2410915372 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f6868055740 con 0x7f687c070d90 2023-11-08T08:19:21.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:21.788 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:21.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.788+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68600682f0 msgr2=0x7f686006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.788+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68600682f0 0x7f686006a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6870001600 tx=0x7f6870007a20 comp rx=0 tx=0).stop 2023-11-08T08:19:21.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.788+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 msgr2=0x7f687c150300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.788+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 0x7f687c150300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f686800c750 tx=0x7f686800cb10 comp rx=0 tx=0).stop 2023-11-08T08:19:21.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 shutdown_connections 2023-11-08T08:19:21.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f68600682f0 0x7f686006a7a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f687c163430 0x7f687c153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f687c06a270 0x7f687c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 --2- 172.21.15.116:0/2410915372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c070d90 0x7f687c150300 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 >> 172.21.15.116:0/2410915372 conn(0x7f687c06c010 msgr2=0x7f687c0b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:21.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 shutdown_connections 2023-11-08T08:19:21.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.789+0000 7f6882dfb700 1 -- 172.21.15.116:0/2410915372 wait complete. 2023-11-08T08:19:21.802 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:21.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.925+0000 7fd7ea6d7700 1 Processor -- start 2023-11-08T08:19:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.925+0000 7fd7ea6d7700 1 -- start start 2023-11-08T08:19:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7ea6d7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7e415a4d0 0x7fd7e406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7ea6d7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e40cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7ea6d7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd7e40cdd50 0x7fd7e40b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7ea6d7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e406ac00 con 0x7fd7e415a4d0 2023-11-08T08:19:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7ea6d7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e415e0d0 con 0x7fd7e40cdd50 2023-11-08T08:19:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7ea6d7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e40ce560 con 0x7fd7e406a7b0 2023-11-08T08:19:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7e415a4d0 0x7fd7e406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7e415a4d0 0x7fd7e406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35388/0 (socket says 172.21.15.116:35388) 2023-11-08T08:19:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 -- 172.21.15.116:0/1025268671 learned_addr learned my addr 172.21.15.116:0/1025268671 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 -- 172.21.15.116:0/1025268671 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 msgr2=0x7fd7e40cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 --2- 172.21.15.116:0/1025268671 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e40cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 -- 172.21.15.116:0/1025268671 >> v2:172.21.15.129:3300/0 conn(0x7fd7e40cdd50 msgr2=0x7fd7e40b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 --2- 172.21.15.116:0/1025268671 >> v2:172.21.15.129:3300/0 conn(0x7fd7e40cdd50 0x7fd7e40b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.926+0000 7fd7e3fff700 1 -- 172.21.15.116:0/1025268671 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e415e320 con 0x7fd7e415a4d0 2023-11-08T08:19:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7e3fff700 1 --2- 172.21.15.116:0/1025268671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e406a270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40af5e0 tx=0x7fd7d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf83c25cc88e9785 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7e2ffd700 1 -- 172.21.15.116:0/1025268671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d800b870 con 0x7fd7e415a4d0 2023-11-08T08:19:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7e2ffd700 1 -- 172.21.15.116:0/1025268671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d800b9d0 con 0x7fd7e415a4d0 2023-11-08T08:19:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7e2ffd700 1 -- 172.21.15.116:0/1025268671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d800bd60 con 0x7fd7e415a4d0 2023-11-08T08:19:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/1025268671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 msgr2=0x7fd7e406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/1025268671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e406a270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40af5e0 tx=0x7fd7d8009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/1025268671 shutdown_connections 2023-11-08T08:19:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/1025268671 >> v2:172.21.15.129:3300/0 conn(0x7fd7e40cdd50 0x7fd7e40b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/1025268671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e406a270 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/1025268671 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e40cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/1025268671 >> 172.21.15.116:0/1025268671 conn(0x7fd7e406c010 msgr2=0x7fd7e40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/1025268671 shutdown_connections 2023-11-08T08:19:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.927+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/1025268671 wait complete. 2023-11-08T08:19:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.928+0000 7fd7ea6d7700 1 Processor -- start 2023-11-08T08:19:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 -- start start 2023-11-08T08:19:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e41f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e40cdd50 0x7fd7e40ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e40ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e40c8180 con 0x7fd7e415a4d0 2023-11-08T08:19:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e406e1b0 con 0x7fd7e40cdd50 2023-11-08T08:19:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7ea6d7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e406e490 con 0x7fd7e406a7b0 2023-11-08T08:19:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7e8c74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e40ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7e8c74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e40ae0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35390/0 (socket says 172.21.15.116:35390) 2023-11-08T08:19:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.929+0000 7fd7e8c74700 1 -- 172.21.15.116:0/4243895585 learned_addr learned my addr 172.21.15.116:0/4243895585 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e3fff700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e41f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e37fe700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e40cdd50 0x7fd7e40ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e8c74700 1 -- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 msgr2=0x7fd7e41f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e8c74700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e41f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e8c74700 1 -- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e40cdd50 msgr2=0x7fd7e40ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e8c74700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e40cdd50 0x7fd7e40ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e8c74700 1 -- 172.21.15.116:0/4243895585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d8009550 con 0x7fd7e415a4d0 2023-11-08T08:19:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e8c74700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e40ae0f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00b970 tx=0x7fd7dc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00c9b0 con 0x7fd7e415a4d0 2023-11-08T08:19:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc00cb30 con 0x7fd7e415a4d0 2023-11-08T08:19:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e415e8e0 con 0x7fd7e415a4d0 2023-11-08T08:19:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc014440 con 0x7fd7e415a4d0 2023-11-08T08:19:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.930+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e415e600 con 0x7fd7e415a4d0 2023-11-08T08:19:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.931+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e415a8a0 con 0x7fd7e415a4d0 2023-11-08T08:19:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.931+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd7dc01b030 con 0x7fd7e415a4d0 2023-11-08T08:19:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.932+0000 7fd7e0ff9700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7d00682f0 0x7fd7d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.932+0000 7fd7e3fff700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7d00682f0 0x7fd7d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.932+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd7dc006e40 con 0x7fd7e415a4d0 2023-11-08T08:19:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.932+0000 7fd7e3fff700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7d00682f0 0x7fd7d006a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8014040 tx=0x7fd7d8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:21.936+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7dc052190 con 0x7fd7e415a4d0 2023-11-08T08:19:22.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.115+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd7e40c82c0 con 0x7fd7e415a4d0 2023-11-08T08:19:22.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.115+0000 7fd7e0ff9700 1 -- 172.21.15.116:0/4243895585 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd7e40c82c0 con 0x7fd7e415a4d0 2023-11-08T08:19:22.117 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:22.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7d00682f0 msgr2=0x7fd7d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7d00682f0 0x7fd7d006a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8014040 tx=0x7fd7d8008030 comp rx=0 tx=0).stop 2023-11-08T08:19:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 msgr2=0x7fd7e40ae0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e40ae0f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00b970 tx=0x7fd7dc00bd30 comp rx=0 tx=0).stop 2023-11-08T08:19:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 shutdown_connections 2023-11-08T08:19:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e415a4d0 0x7fd7e40ae0f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7e406a7b0 0x7fd7e41f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7d00682f0 0x7fd7d006a7a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 --2- 172.21.15.116:0/4243895585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e40cdd50 0x7fd7e40ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 >> 172.21.15.116:0/4243895585 conn(0x7fd7e406c010 msgr2=0x7fd7e414efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 shutdown_connections 2023-11-08T08:19:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.117+0000 7fd7ea6d7700 1 -- 172.21.15.116:0/4243895585 wait complete. 2023-11-08T08:19:22.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:22.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.251+0000 7f35c3aed700 1 Processor -- start 2023-11-08T08:19:22.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.251+0000 7f35c3aed700 1 -- start start 2023-11-08T08:19:22.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c3aed700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35bc15c810 0x7f35bc160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c3aed700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc1607a0 con 0x7f35bc06a270 2023-11-08T08:19:22.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c3aed700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc160a50 con 0x7f35bc0718e0 2023-11-08T08:19:22.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c3aed700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc15cbe0 con 0x7f35bc15c810 2023-11-08T08:19:22.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c208a700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f35bc06a270 0x7f35bc06a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56274/0 (socket says 172.21.15.116:56274) 2023-11-08T08:19:22.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c208a700 1 -- 172.21.15.116:0/282425173 learned_addr learned my addr 172.21.15.116:0/282425173 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:22.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020445434 0 0) 0x7f35bc1607a0 con 0x7f35bc06a270 2023-11-08T08:19:22.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a80035b0 con 0x7f35bc06a270 2023-11-08T08:19:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c1889700 1 --2- 172.21.15.116:0/282425173 >> v2:172.21.15.129:3300/0 conn(0x7f35bc15c810 0x7f35bc160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277077560 0 0) 0x7f35a80035b0 con 0x7f35bc06a270 2023-11-08T08:19:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 >> v2:172.21.15.129:3300/0 conn(0x7f35bc15c810 msgr2=0x7f35bc160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 --2- 172.21.15.116:0/282425173 >> v2:172.21.15.129:3300/0 conn(0x7f35bc15c810 0x7f35bc160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 >> v1:172.21.15.129:6789/0 conn(0x7f35bc0718e0 legacy=0x7f35bc06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc160d30 con 0x7f35bc06a270 2023-11-08T08:19:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f35b40019f0 con 0x7f35bc06a270 2023-11-08T08:19:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 >> v1:172.21.15.116:6789/0 conn(0x7f35bc06a270 legacy=0x7f35bc06a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8003a00 0x7f35a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35a8004850 0x7f35a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35a8005200 0x7f35a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc15cbe0 con 0x7f35a8005200 2023-11-08T08:19:22.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a800c080 con 0x7f35a8004850 2023-11-08T08:19:22.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a800c240 con 0x7f35a8003a00 2023-11-08T08:19:22.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.252+0000 7f35c208a700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35a8005200 0x7f35a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c1088700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35a8004850 0x7f35a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c1889700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8003a00 0x7f35a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 -- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8003a00 msgr2=0x7f35a8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8003a00 0x7f35a8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 -- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35a8004850 msgr2=0x7f35a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35a8004850 0x7f35a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 -- 172.21.15.116:0/282425173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a80035b0 con 0x7f35a8005200 2023-11-08T08:19:22.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c1889700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8003a00 0x7f35a8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:22.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 -- 172.21.15.116:0/282425173 reap_dead start 2023-11-08T08:19:22.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c208a700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35a8005200 0x7f35a800ba10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f35b400a830 tx=0x7f35a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81adc1d7e66d9f5a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b400a2f0 con 0x7f35a8005200 2023-11-08T08:19:22.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b4012620 con 0x7f35a8005200 2023-11-08T08:19:22.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c0887700 1 -- 172.21.15.116:0/282425173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40129d0 con 0x7f35a8005200 2023-11-08T08:19:22.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c3aed700 1 -- 172.21.15.116:0/282425173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35a8005200 msgr2=0x7f35a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.253+0000 7f35c3aed700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35a8005200 0x7f35a800ba10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f35b400a830 tx=0x7f35a8005880 comp rx=0 tx=0).stop 2023-11-08T08:19:22.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.254+0000 7f35c3aed700 1 -- 172.21.15.116:0/282425173 shutdown_connections 2023-11-08T08:19:22.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.254+0000 7f35c3aed700 1 --2- 172.21.15.116:0/282425173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35a8005200 0x7f35a800ba10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.254+0000 7f35c3aed700 1 -- 172.21.15.116:0/282425173 >> 172.21.15.116:0/282425173 conn(0x7f35bc06c010 msgr2=0x7f35bc0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:22.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.254+0000 7f35c3aed700 1 -- 172.21.15.116:0/282425173 shutdown_connections 2023-11-08T08:19:22.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.254+0000 7f35c3aed700 1 -- 172.21.15.116:0/282425173 wait complete. 2023-11-08T08:19:22.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 Processor -- start 2023-11-08T08:19:22.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 -- start start 2023-11-08T08:19:22.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8005200 0x7f35bc0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35bc14ad70 0x7f35bc0b5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 0x7f35bc0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc1f1dd0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0b3390 con 0x7f35bc14ad70 2023-11-08T08:19:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.255+0000 7f35c3aed700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0b3600 con 0x7f35a8005200 2023-11-08T08:19:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 0x7f35bc0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 0x7f35bc0c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35550/0 (socket says 172.21.15.116:35550) 2023-11-08T08:19:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 -- 172.21.15.116:0/38190443 learned_addr learned my addr 172.21.15.116:0/38190443 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c1889700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8005200 0x7f35bc0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c1088700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35bc14ad70 0x7f35bc0b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 -- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8005200 msgr2=0x7f35bc0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8005200 0x7f35bc0b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 -- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35bc14ad70 msgr2=0x7f35bc0b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35bc14ad70 0x7f35bc0b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 -- 172.21.15.116:0/38190443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0ceb20 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c208a700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 0x7f35bc0c8160 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f35b400ae20 tx=0x7f35b4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40124c0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b40019b0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc161010 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b401d660 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.256+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc0ce740 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.257+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc0b3af0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.257+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f35b4010070 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.258+0000 7f35b27fc700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a00682f0 0x7f35a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.258+0000 7f35c1889700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a00682f0 0x7f35a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.258+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f35b4087da0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.258+0000 7f35c1889700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a00682f0 0x7f35a006a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009a90 tx=0x7f35ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.262+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35b4054580 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.440+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35bc0743f0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.440+0000 7f35b27fc700 1 -- 172.21.15.116:0/38190443 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35b4057ba0 con 0x7f35bc0c7cf0 2023-11-08T08:19:22.441 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a00682f0 msgr2=0x7f35a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a00682f0 0x7f35a006a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009a90 tx=0x7f35ac008040 comp rx=0 tx=0).stop 2023-11-08T08:19:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 msgr2=0x7f35bc0c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 0x7f35bc0c8160 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f35b400ae20 tx=0x7f35b4001d80 comp rx=0 tx=0).stop 2023-11-08T08:19:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 shutdown_connections 2023-11-08T08:19:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35bc0c7cf0 0x7f35bc0c8160 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35a8005200 0x7f35bc0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35a00682f0 0x7f35a006a7a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 --2- 172.21.15.116:0/38190443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35bc14ad70 0x7f35bc0b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 >> 172.21.15.116:0/38190443 conn(0x7f35bc06c010 msgr2=0x7f35bc071130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 shutdown_connections 2023-11-08T08:19:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.442+0000 7f35c3aed700 1 -- 172.21.15.116:0/38190443 wait complete. 2023-11-08T08:19:22.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:22.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.579+0000 7efe87a83700 1 Processor -- start 2023-11-08T08:19:22.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.580+0000 7efe87a83700 1 -- start start 2023-11-08T08:19:22.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.580+0000 7efe87a83700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe8006a980 0x7efe800ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.580+0000 7efe87a83700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8015cc10 con 0x7efe800cf710 2023-11-08T08:19:22.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.580+0000 7efe87a83700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8006c9b0 con 0x7efe80071890 2023-11-08T08:19:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.580+0000 7efe87a83700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe8006cc90 con 0x7efe8006a980 2023-11-08T08:19:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efe800cf710 0x7efe8006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56280/0 (socket says 172.21.15.116:56280) 2023-11-08T08:19:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 -- 172.21.15.116:0/172934028 learned_addr learned my addr 172.21.15.116:0/172934028 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561795838 0 0) 0x7efe8015cc10 con 0x7efe800cf710 2023-11-08T08:19:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6c0035b0 con 0x7efe800cf710 2023-11-08T08:19:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8501e700 1 --2- 172.21.15.116:0/172934028 >> v2:172.21.15.129:3300/0 conn(0x7efe8006a980 0x7efe800ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2757205517 0 0) 0x7efe6c0035b0 con 0x7efe800cf710 2023-11-08T08:19:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 >> v2:172.21.15.129:3300/0 conn(0x7efe8006a980 msgr2=0x7efe800ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 --2- 172.21.15.116:0/172934028 >> v2:172.21.15.129:3300/0 conn(0x7efe8006a980 0x7efe800ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 >> v1:172.21.15.129:6789/0 conn(0x7efe80071890 legacy=0x7efe80160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8006cf70 con 0x7efe800cf710 2023-11-08T08:19:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7efe780019f0 con 0x7efe800cf710 2023-11-08T08:19:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 >> v1:172.21.15.116:6789/0 conn(0x7efe800cf710 legacy=0x7efe8006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c003a00 0x7efe6c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6c004850 0x7efe6c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c005200 0x7efe6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8006cc90 con 0x7efe6c003a00 2023-11-08T08:19:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c00c110 con 0x7efe6c004850 2023-11-08T08:19:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c00c2d0 con 0x7efe6c005200 2023-11-08T08:19:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c003a00 0x7efe6c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8501e700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6c004850 0x7efe6c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe86020700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c005200 0x7efe6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 -- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c005200 msgr2=0x7efe6c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c005200 0x7efe6c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 -- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6c004850 msgr2=0x7efe6c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6c004850 0x7efe6c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8581f700 1 -- 172.21.15.116:0/172934028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0035b0 con 0x7efe6c003a00 2023-11-08T08:19:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe86020700 1 -- 172.21.15.116:0/172934028 reap_dead start 2023-11-08T08:19:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.581+0000 7efe8501e700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6c004850 0x7efe6c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe8581f700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c003a00 0x7efe6c003e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efe7800a830 tx=0x7efe6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16f928a024050cc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7800a2f0 con 0x7efe6c003a00 2023-11-08T08:19:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78012600 con 0x7efe6c003a00 2023-11-08T08:19:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe8481d700 1 -- 172.21.15.116:0/172934028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780129b0 con 0x7efe6c003a00 2023-11-08T08:19:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 -- 172.21.15.116:0/172934028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c003a00 msgr2=0x7efe6c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c003a00 0x7efe6c003e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efe7800a830 tx=0x7efe6c005880 comp rx=0 tx=0).stop 2023-11-08T08:19:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 -- 172.21.15.116:0/172934028 shutdown_connections 2023-11-08T08:19:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 --2- 172.21.15.116:0/172934028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c003a00 0x7efe6c003e10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 -- 172.21.15.116:0/172934028 >> 172.21.15.116:0/172934028 conn(0x7efe8006c010 msgr2=0x7efe800cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 -- 172.21.15.116:0/172934028 shutdown_connections 2023-11-08T08:19:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.582+0000 7efe87a83700 1 -- 172.21.15.116:0/172934028 wait complete. 2023-11-08T08:19:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.583+0000 7efe87a83700 1 Processor -- start 2023-11-08T08:19:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.583+0000 7efe87a83700 1 -- start start 2023-11-08T08:19:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe87a83700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c003a00 0x7efe800cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe87a83700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe80150f70 0x7efe801f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe87a83700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 0x7efe800c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe87a83700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800ccb90 con 0x7efe800c9190 2023-11-08T08:19:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe87a83700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800c9b40 con 0x7efe80150f70 2023-11-08T08:19:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe87a83700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015e320 con 0x7efe6c003a00 2023-11-08T08:19:22.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe86020700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 0x7efe800c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe86020700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 0x7efe800c9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35568/0 (socket says 172.21.15.116:35568) 2023-11-08T08:19:22.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe86020700 1 -- 172.21.15.116:0/3907027567 learned_addr learned my addr 172.21.15.116:0/3907027567 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:22.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe8581f700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c003a00 0x7efe800cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.584+0000 7efe8501e700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe80150f70 0x7efe801f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe86020700 1 -- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c003a00 msgr2=0x7efe800cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe86020700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c003a00 0x7efe800cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe86020700 1 -- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe80150f70 msgr2=0x7efe801f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe86020700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe80150f70 0x7efe801f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe86020700 1 -- 172.21.15.116:0/3907027567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8006d530 con 0x7efe800c9190 2023-11-08T08:19:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe86020700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 0x7efe800c9600 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efe8006b2b0 tx=0x7efe7c00e6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c00ee60 con 0x7efe800c9190 2023-11-08T08:19:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c0050d0 con 0x7efe800c9190 2023-11-08T08:19:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8006d250 con 0x7efe800c9190 2023-11-08T08:19:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c005480 con 0x7efe800c9190 2023-11-08T08:19:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.585+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe800ce740 con 0x7efe800c9190 2023-11-08T08:19:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.586+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8015e4b0 con 0x7efe800c9190 2023-11-08T08:19:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.586+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efe7c01b030 con 0x7efe800c9190 2023-11-08T08:19:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.587+0000 7efe767fc700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efe640682f0 0x7efe6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.587+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efe7c084ef0 con 0x7efe800c9190 2023-11-08T08:19:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.587+0000 7efe8581f700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efe640682f0 0x7efe6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.587+0000 7efe8581f700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efe640682f0 0x7efe6406a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efe7800afd0 tx=0x7efe78007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.591+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efe7c0516d0 con 0x7efe800c9190 2023-11-08T08:19:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.770+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe800743f0 con 0x7efe800c9190 2023-11-08T08:19:22.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.771+0000 7efe767fc700 1 -- 172.21.15.116:0/3907027567 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe7c0190a0 con 0x7efe800c9190 2023-11-08T08:19:22.772 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efe640682f0 msgr2=0x7efe6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efe640682f0 0x7efe6406a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efe7800afd0 tx=0x7efe78007000 comp rx=0 tx=0).stop 2023-11-08T08:19:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 msgr2=0x7efe800c9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 0x7efe800c9600 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efe8006b2b0 tx=0x7efe7c00e6f0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 shutdown_connections 2023-11-08T08:19:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800c9190 0x7efe800c9600 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efe6c003a00 0x7efe800cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efe640682f0 0x7efe6406a7a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 --2- 172.21.15.116:0/3907027567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe80150f70 0x7efe801f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 >> 172.21.15.116:0/3907027567 conn(0x7efe8006c010 msgr2=0x7efe8015c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 shutdown_connections 2023-11-08T08:19:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.773+0000 7efe87a83700 1 -- 172.21.15.116:0/3907027567 wait complete. 2023-11-08T08:19:22.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.908+0000 7f44e4ae9700 1 Processor -- start 2023-11-08T08:19:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.908+0000 7f44e4ae9700 1 -- start start 2023-11-08T08:19:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44e4ae9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f44e00b1650 0x7f44e00b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44e4ae9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44e01599b0 0x7f44e015d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44e4ae9700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e0159ec0 con 0x7f44e00af0a0 2023-11-08T08:19:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44e4ae9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e015aa60 con 0x7f44e01599b0 2023-11-08T08:19:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44e4ae9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e00b1f60 con 0x7f44e00b1650 2023-11-08T08:19:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ddd9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44e01599b0 0x7f44e015d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ded9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f44e00af0a0 0x7f44e015a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56294/0 (socket says 172.21.15.116:56294) 2023-11-08T08:19:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ded9d700 1 -- 172.21.15.116:0/191560850 learned_addr learned my addr 172.21.15.116:0/191560850 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44de59c700 1 --2- 172.21.15.116:0/191560850 >> v2:172.21.15.129:3300/0 conn(0x7f44e00b1650 0x7f44e00b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ddd9b700 1 -- 172.21.15.116:0/191560850 >> v2:172.21.15.129:3300/0 conn(0x7f44e00b1650 msgr2=0x7f44e00b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ddd9b700 1 --2- 172.21.15.116:0/191560850 >> v2:172.21.15.129:3300/0 conn(0x7f44e00b1650 0x7f44e00b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ddd9b700 1 -- 172.21.15.116:0/191560850 >> v1:172.21.15.116:6789/0 conn(0x7f44e00af0a0 legacy=0x7f44e015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44ddd9b700 1 -- 172.21.15.116:0/191560850 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e015d8a0 con 0x7f44e01599b0 2023-11-08T08:19:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.909+0000 7f44dd59a700 1 -- 172.21.15.116:0/191560850 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781161215 0 0) 0x7f44e0159ec0 con 0x7f44e00af0a0 2023-11-08T08:19:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.910+0000 7f44ddd9b700 1 --2- 172.21.15.116:0/191560850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e01599b0 0x7f44e015d050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f44e0067310 tx=0x7f44d0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7f775322e9fe89 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.910+0000 7f44dd59a700 1 -- 172.21.15.116:0/191560850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0011830 con 0x7f44e01599b0 2023-11-08T08:19:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.910+0000 7f44dd59a700 1 -- 172.21.15.116:0/191560850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0011990 con 0x7f44e01599b0 2023-11-08T08:19:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.910+0000 7f44dd59a700 1 -- 172.21.15.116:0/191560850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0011d20 con 0x7f44e01599b0 2023-11-08T08:19:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.910+0000 7f44e4ae9700 1 -- 172.21.15.116:0/191560850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e01599b0 msgr2=0x7f44e015d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.910+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/191560850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e01599b0 0x7f44e015d050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f44e0067310 tx=0x7f44d0009b20 comp rx=0 tx=0).stop 2023-11-08T08:19:22.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.911+0000 7f44e4ae9700 1 -- 172.21.15.116:0/191560850 shutdown_connections 2023-11-08T08:19:22.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.911+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/191560850 >> v2:172.21.15.129:3300/0 conn(0x7f44e00b1650 0x7f44e00b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.911+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/191560850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e01599b0 0x7f44e015d050 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.911+0000 7f44e4ae9700 1 -- 172.21.15.116:0/191560850 >> 172.21.15.116:0/191560850 conn(0x7f44e006c010 msgr2=0x7f44e00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:22.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.911+0000 7f44e4ae9700 1 -- 172.21.15.116:0/191560850 shutdown_connections 2023-11-08T08:19:22.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.911+0000 7f44e4ae9700 1 -- 172.21.15.116:0/191560850 wait complete. 2023-11-08T08:19:22.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.912+0000 7f44e4ae9700 1 Processor -- start 2023-11-08T08:19:22.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.912+0000 7f44e4ae9700 1 -- start start 2023-11-08T08:19:22.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44e00af0a0 0x7f44e014bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 0x7f44e00c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44e01490d0 0x7f44e00c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e01467b0 con 0x7f44e00b1650 2023-11-08T08:19:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e0145be0 con 0x7f44e00af0a0 2023-11-08T08:19:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e0145e80 con 0x7f44e01490d0 2023-11-08T08:19:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 0x7f44e00c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 0x7f44e00c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35586/0 (socket says 172.21.15.116:35586) 2023-11-08T08:19:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 -- 172.21.15.116:0/1986106378 learned_addr learned my addr 172.21.15.116:0/1986106378 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44de59c700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44e00af0a0 0x7f44e014bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ded9d700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44e01490d0 0x7f44e00c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 -- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44e01490d0 msgr2=0x7f44e00c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44e01490d0 0x7f44e00c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 -- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44e00af0a0 msgr2=0x7f44e014bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:22.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44e00af0a0 0x7f44e014bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:22.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 -- 172.21.15.116:0/1986106378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00094a0 con 0x7f44e00b1650 2023-11-08T08:19:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44de59c700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44e00af0a0 0x7f44e014bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ddd9b700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 0x7f44e00c58d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f44d0014040 tx=0x7f44d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00038a0 con 0x7f44e00b1650 2023-11-08T08:19:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d001e070 con 0x7f44e00b1650 2023-11-08T08:19:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e015de60 con 0x7f44e00b1650 2023-11-08T08:19:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.913+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0003c20 con 0x7f44e00b1650 2023-11-08T08:19:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.914+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e015db80 con 0x7f44e00b1650 2023-11-08T08:19:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.914+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e015d590 con 0x7f44e00b1650 2023-11-08T08:19:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.915+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f44d000f050 con 0x7f44e00b1650 2023-11-08T08:19:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.915+0000 7f44ceffd700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f44c00682f0 0x7f44c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.915+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f44d0086c60 con 0x7f44e00b1650 2023-11-08T08:19:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.915+0000 7f44de59c700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f44c00682f0 0x7f44c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.916+0000 7f44de59c700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f44c00682f0 0x7f44c006a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009c30 tx=0x7f44d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:22.919+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f44d00534c0 con 0x7f44e00b1650 2023-11-08T08:19:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.098+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44e00743f0 con 0x7f44e00b1650 2023-11-08T08:19:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.099+0000 7f44ceffd700 1 -- 172.21.15.116:0/1986106378 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44d00150a0 con 0x7f44e00b1650 2023-11-08T08:19:23.100 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f44c00682f0 msgr2=0x7f44c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f44c00682f0 0x7f44c006a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009c30 tx=0x7f44d4008040 comp rx=0 tx=0).stop 2023-11-08T08:19:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 msgr2=0x7f44e00c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 0x7f44e00c58d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f44d0014040 tx=0x7f44d0004750 comp rx=0 tx=0).stop 2023-11-08T08:19:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 shutdown_connections 2023-11-08T08:19:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44e01490d0 0x7f44e00c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f44c00682f0 0x7f44c006a7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44e00af0a0 0x7f44e014bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 --2- 172.21.15.116:0/1986106378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e00b1650 0x7f44e00c58d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 >> 172.21.15.116:0/1986106378 conn(0x7f44e006c010 msgr2=0x7f44e00af630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:23.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 shutdown_connections 2023-11-08T08:19:23.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.101+0000 7f44e4ae9700 1 -- 172.21.15.116:0/1986106378 wait complete. 2023-11-08T08:19:23.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.233+0000 7fad5aab2700 1 Processor -- start 2023-11-08T08:19:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.233+0000 7fad5aab2700 1 -- start start 2023-11-08T08:19:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5aab2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fad5415a590 0x7fad5406b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5aab2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad540b50d0 0x7fad5406c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5aab2700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540cd480 con 0x7fad5415a960 2023-11-08T08:19:23.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5aab2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540cd760 con 0x7fad540b50d0 2023-11-08T08:19:23.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5aab2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540cda40 con 0x7fad5415a590 2023-11-08T08:19:23.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5904f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fad5415a960 0x7fad5415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56304/0 (socket says 172.21.15.116:56304) 2023-11-08T08:19:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad53fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad540b50d0 0x7fad5406c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.234+0000 7fad5904f700 1 -- 172.21.15.116:0/3746401900 learned_addr learned my addr 172.21.15.116:0/3746401900 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad5884e700 1 --2- 172.21.15.116:0/3746401900 >> v2:172.21.15.129:3300/0 conn(0x7fad5415a590 0x7fad5406b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad53fff700 1 -- 172.21.15.116:0/3746401900 >> v2:172.21.15.129:3300/0 conn(0x7fad5415a590 msgr2=0x7fad5406b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad53fff700 1 --2- 172.21.15.116:0/3746401900 >> v2:172.21.15.129:3300/0 conn(0x7fad5415a590 0x7fad5406b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad53fff700 1 -- 172.21.15.116:0/3746401900 >> v1:172.21.15.116:6789/0 conn(0x7fad5415a960 legacy=0x7fad5415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad53fff700 1 -- 172.21.15.116:0/3746401900 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540cdd20 con 0x7fad540b50d0 2023-11-08T08:19:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad537fe700 1 -- 172.21.15.116:0/3746401900 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356907447 0 0) 0x7fad540cd480 con 0x7fad5415a960 2023-11-08T08:19:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad5884e700 1 --2- 172.21.15.116:0/3746401900 >> v2:172.21.15.129:3300/0 conn(0x7fad5415a590 0x7fad5406b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad53fff700 1 --2- 172.21.15.116:0/3746401900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540b50d0 0x7fad5406c440 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fad540cceb0 tx=0x7fad40009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885f315c21c8b005 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad537fe700 1 -- 172.21.15.116:0/3746401900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011830 con 0x7fad540b50d0 2023-11-08T08:19:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad537fe700 1 -- 172.21.15.116:0/3746401900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40011990 con 0x7fad540b50d0 2023-11-08T08:19:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.235+0000 7fad537fe700 1 -- 172.21.15.116:0/3746401900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011d20 con 0x7fad540b50d0 2023-11-08T08:19:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 -- 172.21.15.116:0/3746401900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540b50d0 msgr2=0x7fad5406c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3746401900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540b50d0 0x7fad5406c440 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fad540cceb0 tx=0x7fad40009b20 comp rx=0 tx=0).stop 2023-11-08T08:19:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 -- 172.21.15.116:0/3746401900 shutdown_connections 2023-11-08T08:19:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3746401900 >> v2:172.21.15.129:3300/0 conn(0x7fad5415a590 0x7fad5406b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3746401900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540b50d0 0x7fad5406c440 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 -- 172.21.15.116:0/3746401900 >> 172.21.15.116:0/3746401900 conn(0x7fad540cf400 msgr2=0x7fad540cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 -- 172.21.15.116:0/3746401900 shutdown_connections 2023-11-08T08:19:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.236+0000 7fad5aab2700 1 -- 172.21.15.116:0/3746401900 wait complete. 2023-11-08T08:19:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.237+0000 7fad5aab2700 1 Processor -- start 2023-11-08T08:19:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.237+0000 7fad5aab2700 1 -- start start 2023-11-08T08:19:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad540b50d0 0x7fad54156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad5415a590 0x7fad54155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 0x7fad54154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad541f3880 con 0x7fad5415a960 2023-11-08T08:19:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad541f3b60 con 0x7fad540b50d0 2023-11-08T08:19:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad541f3e40 con 0x7fad5415a590 2023-11-08T08:19:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 0x7fad54154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 0x7fad54154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35616/0 (socket says 172.21.15.116:35616) 2023-11-08T08:19:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 -- 172.21.15.116:0/3124091972 learned_addr learned my addr 172.21.15.116:0/3124091972 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5884e700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad540b50d0 0x7fad54156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad53fff700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad5415a590 0x7fad54155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 -- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad5415a590 msgr2=0x7fad54155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad5415a590 0x7fad54155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 -- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad540b50d0 msgr2=0x7fad54156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad540b50d0 0x7fad54156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 -- 172.21.15.116:0/3124091972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400094a0 con 0x7fad5415a960 2023-11-08T08:19:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5884e700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad540b50d0 0x7fad54156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5904f700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 0x7fad54154450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00cba0 tx=0x7fad4c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00ab80 con 0x7fad5415a960 2023-11-08T08:19:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00ad00 con 0x7fad5415a960 2023-11-08T08:19:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5406baf0 con 0x7fad5415a960 2023-11-08T08:19:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c007310 con 0x7fad5415a960 2023-11-08T08:19:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.238+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad540ce000 con 0x7fad5415a960 2023-11-08T08:19:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.239+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5406c980 con 0x7fad5415a960 2023-11-08T08:19:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.240+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad4c01a030 con 0x7fad5415a960 2023-11-08T08:19:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.240+0000 7fad517fa700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad380682f0 0x7fad3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.240+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fad4c0859e0 con 0x7fad5415a960 2023-11-08T08:19:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.240+0000 7fad5884e700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad380682f0 0x7fad3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.240+0000 7fad5884e700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad380682f0 0x7fad3806a7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fad48009a50 tx=0x7fad48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.244+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad4c0521e0 con 0x7fad5415a960 2023-11-08T08:19:23.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.438+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fad540743f0 con 0x7fad5415a960 2023-11-08T08:19:23.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.439+0000 7fad517fa700 1 -- 172.21.15.116:0/3124091972 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fad4c018080 con 0x7fad5415a960 2023-11-08T08:19:23.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.440+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad380682f0 msgr2=0x7fad3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.440+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad380682f0 0x7fad3806a7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fad48009a50 tx=0x7fad48008040 comp rx=0 tx=0).stop 2023-11-08T08:19:23.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.440+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 msgr2=0x7fad54154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.440+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 0x7fad54154450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00cba0 tx=0x7fad4c00cf60 comp rx=0 tx=0).stop 2023-11-08T08:19:23.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.440+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 shutdown_connections 2023-11-08T08:19:23.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5415a960 0x7fad54154450 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad380682f0 0x7fad3806a7a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad540b50d0 0x7fad54156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 --2- 172.21.15.116:0/3124091972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad5415a590 0x7fad54155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 >> 172.21.15.116:0/3124091972 conn(0x7fad540cf400 msgr2=0x7fad5406cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:23.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 shutdown_connections 2023-11-08T08:19:23.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.441+0000 7fad5aab2700 1 -- 172.21.15.116:0/3124091972 wait complete. 2023-11-08T08:19:23.445 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:23.453 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:23.454 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:23.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:23.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.630+0000 7ff09e34b700 1 Processor -- start 2023-11-08T08:19:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.630+0000 7ff09e34b700 1 -- start start 2023-11-08T08:19:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09e34b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09815cc80 0x7ff09806de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09e34b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff09806e3a0 0x7ff098070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09e34b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0980b1810 con 0x7ff09806a270 2023-11-08T08:19:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09e34b700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0980b2490 con 0x7ff09806e3a0 2023-11-08T08:19:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09e34b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0980b1af0 con 0x7ff09815cc80 2023-11-08T08:19:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09c8e8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff09806e3a0 0x7ff098070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09c8e8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff09806e3a0 0x7ff098070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35632/0 (socket says 172.21.15.116:35632) 2023-11-08T08:19:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff09c8e8700 1 -- 172.21.15.116:0/3331039399 learned_addr learned my addr 172.21.15.116:0/3331039399 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.631+0000 7ff0977fe700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09815cc80 0x7ff09806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09c8e8700 1 -- 172.21.15.116:0/3331039399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09815cc80 msgr2=0x7ff09806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09c8e8700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09815cc80 0x7ff09806de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09c8e8700 1 -- 172.21.15.116:0/3331039399 >> v1:172.21.15.129:6789/0 conn(0x7ff09806a270 legacy=0x7ff09806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09c8e8700 1 -- 172.21.15.116:0/3331039399 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0980b1dd0 con 0x7ff09806e3a0 2023-11-08T08:19:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff0977fe700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09815cc80 0x7ff09806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09c8e8700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff098070d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff090009e50 tx=0x7ff09000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce947b5fb24b22b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff096ffd700 1 -- 172.21.15.116:0/3331039399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09000e070 con 0x7ff09806e3a0 2023-11-08T08:19:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff096ffd700 1 -- 172.21.15.116:0/3331039399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff090003a20 con 0x7ff09806e3a0 2023-11-08T08:19:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff096ffd700 1 -- 172.21.15.116:0/3331039399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090003dd0 con 0x7ff09806e3a0 2023-11-08T08:19:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 -- 172.21.15.116:0/3331039399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 msgr2=0x7ff098070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff098070d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff090009e50 tx=0x7ff09000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 -- 172.21.15.116:0/3331039399 shutdown_connections 2023-11-08T08:19:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff098070d20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 --2- 172.21.15.116:0/3331039399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09815cc80 0x7ff09806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 -- 172.21.15.116:0/3331039399 >> 172.21.15.116:0/3331039399 conn(0x7ff09806c010 msgr2=0x7ff0980cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 -- 172.21.15.116:0/3331039399 shutdown_connections 2023-11-08T08:19:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.632+0000 7ff09e34b700 1 -- 172.21.15.116:0/3331039399 wait complete. 2023-11-08T08:19:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.634+0000 7ff09e34b700 1 Processor -- start 2023-11-08T08:19:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.634+0000 7ff09e34b700 1 -- start start 2023-11-08T08:19:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09e34b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09806a270 0x7ff098152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09e34b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff0980c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09e34b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff09815cc80 0x7ff0981525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09e34b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0980b4200 con 0x7ff09806e3a0 2023-11-08T08:19:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09e34b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0980c91a0 con 0x7ff09815cc80 2023-11-08T08:19:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09e34b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff098152b10 con 0x7ff09806a270 2023-11-08T08:19:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff0980c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff0980c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35640/0 (socket says 172.21.15.116:35640) 2023-11-08T08:19:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 -- 172.21.15.116:0/1701049529 learned_addr learned my addr 172.21.15.116:0/1701049529 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff097fff700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09806a270 0x7ff098152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff09c8e8700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff09815cc80 0x7ff0981525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 -- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09806a270 msgr2=0x7ff098152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09806a270 0x7ff098152d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 -- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff09815cc80 msgr2=0x7ff0981525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff09815cc80 0x7ff0981525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.635+0000 7ff0977fe700 1 -- 172.21.15.116:0/1701049529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff090009b30 con 0x7ff09806e3a0 2023-11-08T08:19:23.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.636+0000 7ff0977fe700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff0980c8c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c820 tx=0x7ff08800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.636+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088011800 con 0x7ff09806e3a0 2023-11-08T08:19:23.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.636+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088011960 con 0x7ff09806e3a0 2023-11-08T08:19:23.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.636+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09806e7e0 con 0x7ff09806e3a0 2023-11-08T08:19:23.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.636+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088011d10 con 0x7ff09806e3a0 2023-11-08T08:19:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.636+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0980b20b0 con 0x7ff09806e3a0 2023-11-08T08:19:23.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.637+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09815daa0 con 0x7ff09806e3a0 2023-11-08T08:19:23.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.637+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff088021070 con 0x7ff09806e3a0 2023-11-08T08:19:23.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.638+0000 7ff094ff9700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff0780682f0 0x7ff07806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.638+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff088085b90 con 0x7ff09806e3a0 2023-11-08T08:19:23.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.638+0000 7ff097fff700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff0780682f0 0x7ff07806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.638+0000 7ff097fff700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff0780682f0 0x7ff07806a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0981f5c70 tx=0x7ff080008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.642+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff088052370 con 0x7ff09806e3a0 2023-11-08T08:19:23.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.823+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0980b4340 con 0x7ff09806e3a0 2023-11-08T08:19:23.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.824+0000 7ff094ff9700 1 -- 172.21.15.116:0/1701049529 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7ff0980b4340 con 0x7ff09806e3a0 2023-11-08T08:19:23.826 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:23.826 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:23.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff0780682f0 msgr2=0x7ff07806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff0780682f0 0x7ff07806a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0981f5c70 tx=0x7ff080008040 comp rx=0 tx=0).stop 2023-11-08T08:19:23.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 msgr2=0x7ff0980c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff0980c8c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c820 tx=0x7ff08800cbe0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 shutdown_connections 2023-11-08T08:19:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff0780682f0 0x7ff07806a7a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff09815cc80 0x7ff0981525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff09806a270 0x7ff098152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 --2- 172.21.15.116:0/1701049529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09806e3a0 0x7ff0980c8c60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.826+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 >> 172.21.15.116:0/1701049529 conn(0x7ff09806c010 msgr2=0x7ff09815a830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:23.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.827+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 shutdown_connections 2023-11-08T08:19:23.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.827+0000 7ff09e34b700 1 -- 172.21.15.116:0/1701049529 wait complete. 2023-11-08T08:19:23.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:23.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:23.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.962+0000 7fda92dc2700 1 Processor -- start 2023-11-08T08:19:23.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.962+0000 7fda92dc2700 1 -- start start 2023-11-08T08:19:23.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda92dc2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fda8c1599b0 0x7fda8c15d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda92dc2700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c159ec0 con 0x7fda8c0b1650 2023-11-08T08:19:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda92dc2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c15aa60 con 0x7fda8c0af0a0 2023-11-08T08:19:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda92dc2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c15d8a0 con 0x7fda8c1599b0 2023-11-08T08:19:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda90b5e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fda8c0b1650 0x7fda8c0b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56320/0 (socket says 172.21.15.116:56320) 2023-11-08T08:19:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda90b5e700 1 -- 172.21.15.116:0/3419562075 learned_addr learned my addr 172.21.15.116:0/3419562075 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8bfff700 1 --2- 172.21.15.116:0/3419562075 >> v2:172.21.15.129:3300/0 conn(0x7fda8c1599b0 0x7fda8c15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070348458 0 0) 0x7fda8c159ec0 con 0x7fda8c0b1650 2023-11-08T08:19:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda7c0035b0 con 0x7fda8c0b1650 2023-11-08T08:19:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2612224236 0 0) 0x7fda7c0035b0 con 0x7fda8c0b1650 2023-11-08T08:19:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 >> v2:172.21.15.129:3300/0 conn(0x7fda8c1599b0 msgr2=0x7fda8c15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 --2- 172.21.15.116:0/3419562075 >> v2:172.21.15.129:3300/0 conn(0x7fda8c1599b0 0x7fda8c15d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 >> v1:172.21.15.129:6789/0 conn(0x7fda8c0af0a0 legacy=0x7fda8c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c15db80 con 0x7fda8c0b1650 2023-11-08T08:19:23.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fda80001670 con 0x7fda8c0b1650 2023-11-08T08:19:23.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 >> v1:172.21.15.116:6789/0 conn(0x7fda8c0b1650 legacy=0x7fda8c0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c003a00 0x7fda7c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda7c004850 0x7fda7c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda7c005200 0x7fda7c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c15d8a0 con 0x7fda7c005200 2023-11-08T08:19:23.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c00c080 con 0x7fda7c004850 2023-11-08T08:19:23.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.963+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c00c240 con 0x7fda7c003a00 2023-11-08T08:19:23.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda7c005200 0x7fda7c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda8bfff700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda7c004850 0x7fda7c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda90b5e700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c003a00 0x7fda7c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 -- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c003a00 msgr2=0x7fda7c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:23.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c003a00 0x7fda7c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 -- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda7c004850 msgr2=0x7fda7c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:23.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda7c004850 0x7fda7c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 -- 172.21.15.116:0/3419562075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0035b0 con 0x7fda7c005200 2023-11-08T08:19:23.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 -- 172.21.15.116:0/3419562075 reap_dead start 2023-11-08T08:19:23.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda9135f700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda7c005200 0x7fda7c00ba10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c190 tx=0x7fda7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1193c61bb4afeb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84001db0 con 0x7fda7c005200 2023-11-08T08:19:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8400ccf0 con 0x7fda7c005200 2023-11-08T08:19:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.964+0000 7fda8b7fe700 1 -- 172.21.15.116:0/3419562075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84003c40 con 0x7fda7c005200 2023-11-08T08:19:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 -- 172.21.15.116:0/3419562075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda7c005200 msgr2=0x7fda7c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda7c005200 0x7fda7c00ba10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c190 tx=0x7fda7c005880 comp rx=0 tx=0).stop 2023-11-08T08:19:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 -- 172.21.15.116:0/3419562075 shutdown_connections 2023-11-08T08:19:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 --2- 172.21.15.116:0/3419562075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda7c005200 0x7fda7c00ba10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 -- 172.21.15.116:0/3419562075 >> 172.21.15.116:0/3419562075 conn(0x7fda8c06c010 msgr2=0x7fda8c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 -- 172.21.15.116:0/3419562075 shutdown_connections 2023-11-08T08:19:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.965+0000 7fda92dc2700 1 -- 172.21.15.116:0/3419562075 wait complete. 2023-11-08T08:19:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.966+0000 7fda92dc2700 1 Processor -- start 2023-11-08T08:19:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.966+0000 7fda92dc2700 1 -- start start 2023-11-08T08:19:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c005200 0x7fda8c1f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c1f2b90 0x7fda8c15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 0x7fda8c070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c1f33a0 con 0x7fda8c153ef0 2023-11-08T08:19:23.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c1f3680 con 0x7fda8c1f2b90 2023-11-08T08:19:23.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c1f3960 con 0x7fda7c005200 2023-11-08T08:19:23.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 0x7fda8c070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 0x7fda8c070bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35658/0 (socket says 172.21.15.116:35658) 2023-11-08T08:19:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 -- 172.21.15.116:0/2913289681 learned_addr learned my addr 172.21.15.116:0/2913289681 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda90b5e700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c005200 0x7fda8c1f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda8bfff700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c1f2b90 0x7fda8c15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 -- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c005200 msgr2=0x7fda8c1f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c005200 0x7fda8c1f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 -- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c1f2b90 msgr2=0x7fda8c15b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:23.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c1f2b90 0x7fda8c15b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:23.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 -- 172.21.15.116:0/2913289681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c15a350 con 0x7fda8c153ef0 2023-11-08T08:19:23.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda9135f700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 0x7fda8c070bd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fda84005040 tx=0x7fda840014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840052d0 con 0x7fda8c153ef0 2023-11-08T08:19:23.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda84005430 con 0x7fda8c153ef0 2023-11-08T08:19:23.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c15de60 con 0x7fda8c153ef0 2023-11-08T08:19:23.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840057a0 con 0x7fda8c153ef0 2023-11-08T08:19:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.967+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c15a4e0 con 0x7fda8c153ef0 2023-11-08T08:19:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.968+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c068450 con 0x7fda8c153ef0 2023-11-08T08:19:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.969+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda84026030 con 0x7fda8c153ef0 2023-11-08T08:19:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.969+0000 7fda897fa700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda700682f0 0x7fda7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.969+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fda840866b0 con 0x7fda8c153ef0 2023-11-08T08:19:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.969+0000 7fda90b5e700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda700682f0 0x7fda7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.970+0000 7fda90b5e700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda700682f0 0x7fda7006a7a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fda80000c00 tx=0x7fda80007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:23.973+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda84052f60 con 0x7fda8c153ef0 2023-11-08T08:19:24.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.153+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fda8c156170 con 0x7fda8c153ef0 2023-11-08T08:19:24.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.153+0000 7fda897fa700 1 -- 172.21.15.116:0/2913289681 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fda84056580 con 0x7fda8c153ef0 2023-11-08T08:19:24.155 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:24.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda700682f0 msgr2=0x7fda7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda700682f0 0x7fda7006a7a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fda80000c00 tx=0x7fda80007040 comp rx=0 tx=0).stop 2023-11-08T08:19:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 msgr2=0x7fda8c070bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 0x7fda8c070bd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fda84005040 tx=0x7fda840014a0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 shutdown_connections 2023-11-08T08:19:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c153ef0 0x7fda8c070bd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda7c005200 0x7fda8c1f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda700682f0 0x7fda7006a7a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 --2- 172.21.15.116:0/2913289681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c1f2b90 0x7fda8c15b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 >> 172.21.15.116:0/2913289681 conn(0x7fda8c06c010 msgr2=0x7fda8c0b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 shutdown_connections 2023-11-08T08:19:24.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.155+0000 7fda92dc2700 1 -- 172.21.15.116:0/2913289681 wait complete. 2023-11-08T08:19:24.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:24.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.290+0000 7f9bf1e52700 1 Processor -- start 2023-11-08T08:19:24.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.290+0000 7f9bf1e52700 1 -- start start 2023-11-08T08:19:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.290+0000 7f9bf1e52700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9bec15a5f0 0x7f9bec0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.290+0000 7f9bf1e52700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec0b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.290+0000 7f9bf1e52700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9bec0b2d10 0x7f9bec15e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bf1e52700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec0b3190 con 0x7f9bec15a5f0 2023-11-08T08:19:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bf1e52700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec15a9c0 con 0x7f9bec0b2d10 2023-11-08T08:19:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bf1e52700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0c7b80 con 0x7f9bec0cc200 2023-11-08T08:19:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9bec15a5f0 0x7f9bec0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9bec15a5f0 0x7f9bec0d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35660/0 (socket says 172.21.15.116:35660) 2023-11-08T08:19:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 -- 172.21.15.116:0/420432635 learned_addr learned my addr 172.21.15.116:0/420432635 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bebfff700 1 --2- 172.21.15.116:0/420432635 >> v2:172.21.15.129:3300/0 conn(0x7f9bec0b2d10 0x7f9bec15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beaffd700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec0b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 -- 172.21.15.116:0/420432635 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 msgr2=0x7f9bec0b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec0b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 -- 172.21.15.116:0/420432635 >> v2:172.21.15.129:3300/0 conn(0x7f9bec0b2d10 msgr2=0x7f9bec15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 --2- 172.21.15.116:0/420432635 >> v2:172.21.15.129:3300/0 conn(0x7f9bec0b2d10 0x7f9bec15e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 -- 172.21.15.116:0/420432635 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec0cc990 con 0x7f9bec15a5f0 2023-11-08T08:19:24.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beaffd700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec0b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:24.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9beb7fe700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0d0ad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9bec073cf0 tx=0x7f9bdc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e15434033cbe4d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bea7fc700 1 -- 172.21.15.116:0/420432635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc011830 con 0x7f9bec15a5f0 2023-11-08T08:19:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bea7fc700 1 -- 172.21.15.116:0/420432635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc011990 con 0x7f9bec15a5f0 2023-11-08T08:19:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.291+0000 7f9bea7fc700 1 -- 172.21.15.116:0/420432635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc011d20 con 0x7f9bec15a5f0 2023-11-08T08:19:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 -- 172.21.15.116:0/420432635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 msgr2=0x7f9bec0d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0d0ad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9bec073cf0 tx=0x7f9bdc009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 -- 172.21.15.116:0/420432635 shutdown_connections 2023-11-08T08:19:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/420432635 >> v2:172.21.15.129:3300/0 conn(0x7f9bec0b2d10 0x7f9bec15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0d0ad0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/420432635 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec0b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 -- 172.21.15.116:0/420432635 >> 172.21.15.116:0/420432635 conn(0x7f9bec06ea80 msgr2=0x7f9bec06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 -- 172.21.15.116:0/420432635 shutdown_connections 2023-11-08T08:19:24.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.292+0000 7f9bf1e52700 1 -- 172.21.15.116:0/420432635 wait complete. 2023-11-08T08:19:24.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.293+0000 7f9bf1e52700 1 Processor -- start 2023-11-08T08:19:24.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.293+0000 7f9bf1e52700 1 -- start start 2023-11-08T08:19:24.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bf1e52700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b2d10 0x7f9bec06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bf1e52700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec1f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bf1e52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bf1e52700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec1f32a0 con 0x7f9bec15a5f0 2023-11-08T08:19:24.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bf1e52700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec14cc60 con 0x7f9bec0b2d10 2023-11-08T08:19:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bf1e52700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec06c330 con 0x7f9bec0cc200 2023-11-08T08:19:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0b0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35666/0 (socket says 172.21.15.116:35666) 2023-11-08T08:19:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 -- 172.21.15.116:0/159834164 learned_addr learned my addr 172.21.15.116:0/159834164 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 -- 172.21.15.116:0/159834164 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 msgr2=0x7f9bec1f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec1f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 -- 172.21.15.116:0/159834164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b2d10 msgr2=0x7f9bec06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b2d10 0x7f9bec06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 -- 172.21.15.116:0/159834164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc009550 con 0x7f9bec15a5f0 2023-11-08T08:19:24.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bebfff700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0b0e10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9be400efc0 tx=0x7f9be400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.294+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4009ba0 con 0x7f9bec15a5f0 2023-11-08T08:19:24.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.295+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be400f040 con 0x7f9bec15a5f0 2023-11-08T08:19:24.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.295+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec0ccf50 con 0x7f9bec15a5f0 2023-11-08T08:19:24.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.295+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4014480 con 0x7f9bec15a5f0 2023-11-08T08:19:24.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.295+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec0ccc70 con 0x7f9bec15a5f0 2023-11-08T08:19:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.295+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec0cd230 con 0x7f9bec15a5f0 2023-11-08T08:19:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.296+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9be40145e0 con 0x7f9bec15a5f0 2023-11-08T08:19:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.296+0000 7f9bd3fff700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9bd80682f0 0x7f9bd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.296+0000 7f9beb7fe700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9bd80682f0 0x7f9bd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.296+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9be4086690 con 0x7f9bec15a5f0 2023-11-08T08:19:24.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.297+0000 7f9beb7fe700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9bd80682f0 0x7f9bd806a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009130 tx=0x7f9bdc010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.301+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9be4052ef0 con 0x7f9bec15a5f0 2023-11-08T08:19:24.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.481+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bec0743f0 con 0x7f9bec15a5f0 2023-11-08T08:19:24.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.481+0000 7f9bd3fff700 1 -- 172.21.15.116:0/159834164 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9be4056510 con 0x7f9bec15a5f0 2023-11-08T08:19:24.483 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:24.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9bd80682f0 msgr2=0x7f9bd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9bd80682f0 0x7f9bd806a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009130 tx=0x7f9bdc010000 comp rx=0 tx=0).stop 2023-11-08T08:19:24.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 msgr2=0x7f9bec0b0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0b0e10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9be400efc0 tx=0x7f9be400c3a0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 shutdown_connections 2023-11-08T08:19:24.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec15a5f0 0x7f9bec0b0e10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9bd80682f0 0x7f9bd806a7a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b2d10 0x7f9bec06b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 --2- 172.21.15.116:0/159834164 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9bec0cc200 0x7f9bec1f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 >> 172.21.15.116:0/159834164 conn(0x7f9bec06ea80 msgr2=0x7f9bec0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:24.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 shutdown_connections 2023-11-08T08:19:24.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.483+0000 7f9bf1e52700 1 -- 172.21.15.116:0/159834164 wait complete. 2023-11-08T08:19:24.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:24.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.619+0000 7f263bb6e700 1 Processor -- start 2023-11-08T08:19:24.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.619+0000 7f263bb6e700 1 -- start start 2023-11-08T08:19:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263bb6e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f263415fe50 0x7f263406d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263bb6e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634161370 con 0x7f26340b27d0 2023-11-08T08:19:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263bb6e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26340cf700 con 0x7f26340b3b80 2023-11-08T08:19:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263bb6e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2634160220 con 0x7f263415fe50 2023-11-08T08:19:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263a10b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f26340b27d0 0x7f26340cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56332/0 (socket says 172.21.15.116:56332) 2023-11-08T08:19:24.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263a10b700 1 -- 172.21.15.116:0/1088758863 learned_addr learned my addr 172.21.15.116:0/1088758863 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:24.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.620+0000 7f263990a700 1 --2- 172.21.15.116:0/1088758863 >> v2:172.21.15.129:3300/0 conn(0x7f263415fe50 0x7f263406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255573828 0 0) 0x7f2634161370 con 0x7f26340b27d0 2023-11-08T08:19:24.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26200035b0 con 0x7f26340b27d0 2023-11-08T08:19:24.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166117973 0 0) 0x7f26200035b0 con 0x7f26340b27d0 2023-11-08T08:19:24.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 >> v2:172.21.15.129:3300/0 conn(0x7f263415fe50 msgr2=0x7f263406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 --2- 172.21.15.116:0/1088758863 >> v2:172.21.15.129:3300/0 conn(0x7f263415fe50 0x7f263406d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 >> v1:172.21.15.129:6789/0 conn(0x7f26340b3b80 legacy=0x7f2634160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340b3fd0 con 0x7f26340b27d0 2023-11-08T08:19:24.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f262c0019f0 con 0x7f26340b27d0 2023-11-08T08:19:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 >> v1:172.21.15.116:6789/0 conn(0x7f26340b27d0 legacy=0x7f26340cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2620003a00 0x7f2620003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620004850 0x7f2620004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2620005200 0x7f262000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634160220 con 0x7f2620003a00 2023-11-08T08:19:24.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262000c080 con 0x7f2620005200 2023-11-08T08:19:24.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262000c240 con 0x7f2620004850 2023-11-08T08:19:24.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2620003a00 0x7f2620003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263a10b700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2620005200 0x7f262000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f2639109700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620004850 0x7f2620004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 -- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620004850 msgr2=0x7f2620004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620004850 0x7f2620004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 -- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2620005200 msgr2=0x7f262000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2620005200 0x7f262000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 -- 172.21.15.116:0/1088758863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200035b0 con 0x7f2620003a00 2023-11-08T08:19:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263a10b700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2620005200 0x7f262000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263a10b700 1 -- 172.21.15.116:0/1088758863 reap_dead start 2023-11-08T08:19:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.621+0000 7f263990a700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2620003a00 0x7f2620003e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2624009b60 tx=0x7f2620005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a977627d93eb6bcc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262400e6c0 con 0x7f2620003a00 2023-11-08T08:19:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262400e820 con 0x7f2620003a00 2023-11-08T08:19:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f2638908700 1 -- 172.21.15.116:0/1088758863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262400eb90 con 0x7f2620003a00 2023-11-08T08:19:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 -- 172.21.15.116:0/1088758863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2620003a00 msgr2=0x7f2620003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2620003a00 0x7f2620003e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2624009b60 tx=0x7f2620005880 comp rx=0 tx=0).stop 2023-11-08T08:19:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 -- 172.21.15.116:0/1088758863 shutdown_connections 2023-11-08T08:19:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 --2- 172.21.15.116:0/1088758863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2620003a00 0x7f2620003e10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 -- 172.21.15.116:0/1088758863 >> 172.21.15.116:0/1088758863 conn(0x7f263406ea80 msgr2=0x7f263406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 -- 172.21.15.116:0/1088758863 shutdown_connections 2023-11-08T08:19:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.622+0000 7f263bb6e700 1 -- 172.21.15.116:0/1088758863 wait complete. 2023-11-08T08:19:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.623+0000 7f263bb6e700 1 Processor -- start 2023-11-08T08:19:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.623+0000 7f263bb6e700 1 -- start start 2023-11-08T08:19:24.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263bb6e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620003a00 0x7f26340ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263bb6e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 0x7f26340cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263bb6e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f263415e1e0 0x7f2634159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263bb6e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263415b2b0 con 0x7f263415aa60 2023-11-08T08:19:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263bb6e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340c9630 con 0x7f263415e1e0 2023-11-08T08:19:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263bb6e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26340c9910 con 0x7f2620003a00 2023-11-08T08:19:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 0x7f26340cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 0x7f26340cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35676/0 (socket says 172.21.15.116:35676) 2023-11-08T08:19:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 -- 172.21.15.116:0/4206066633 learned_addr learned my addr 172.21.15.116:0/4206066633 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263990a700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620003a00 0x7f26340ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263a10b700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f263415e1e0 0x7f2634159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 -- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620003a00 msgr2=0x7f26340ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620003a00 0x7f26340ca850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 -- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f263415e1e0 msgr2=0x7f2634159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f263415e1e0 0x7f2634159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f2639109700 1 -- 172.21.15.116:0/4206066633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340cf0b0 con 0x7f263415aa60 2023-11-08T08:19:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263990a700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620003a00 0x7f26340ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.624+0000 7f263a10b700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f263415e1e0 0x7f2634159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.625+0000 7f2639109700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 0x7f26340cbb10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f263000e550 tx=0x7f263000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.625+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300052e0 con 0x7f263415aa60 2023-11-08T08:19:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.625+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2630005440 con 0x7f263415aa60 2023-11-08T08:19:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.625+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26340cedd0 con 0x7f263415aa60 2023-11-08T08:19:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.625+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300038a0 con 0x7f263415aa60 2023-11-08T08:19:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.625+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26340cf390 con 0x7f263415aa60 2023-11-08T08:19:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.626+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26340743f0 con 0x7f263415aa60 2023-11-08T08:19:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.626+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f263001b030 con 0x7f263415aa60 2023-11-08T08:19:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.626+0000 7f262a7fc700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26180682f0 0x7f261806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.627+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2630085220 con 0x7f263415aa60 2023-11-08T08:19:24.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.627+0000 7f263990a700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26180682f0 0x7f261806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.627+0000 7f263990a700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26180682f0 0x7f261806a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f26240090b0 tx=0x7f2624008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.631+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f263004d5e0 con 0x7f263415aa60 2023-11-08T08:19:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.811+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f263406ad90 con 0x7f263415aa60 2023-11-08T08:19:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.811+0000 7f262a7fc700 1 -- 172.21.15.116:0/4206066633 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2630051b30 con 0x7f263415aa60 2023-11-08T08:19:24.813 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.813+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26180682f0 msgr2=0x7f261806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.813+0000 7f263bb6e700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26180682f0 0x7f261806a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f26240090b0 tx=0x7f2624008030 comp rx=0 tx=0).stop 2023-11-08T08:19:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.813+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 msgr2=0x7f26340cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.813+0000 7f263bb6e700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 0x7f26340cbb10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f263000e550 tx=0x7f263000e910 comp rx=0 tx=0).stop 2023-11-08T08:19:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 shutdown_connections 2023-11-08T08:19:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f26180682f0 0x7f261806a7a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f263415e1e0 0x7f2634159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2620003a00 0x7f26340ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 --2- 172.21.15.116:0/4206066633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f263415aa60 0x7f26340cbb10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 >> 172.21.15.116:0/4206066633 conn(0x7f263406ea80 msgr2=0x7f26340d0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 shutdown_connections 2023-11-08T08:19:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.814+0000 7f263bb6e700 1 -- 172.21.15.116:0/4206066633 wait complete. 2023-11-08T08:19:24.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.949+0000 7f3810aca700 1 Processor -- start 2023-11-08T08:19:24.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.949+0000 7f3810aca700 1 -- start start 2023-11-08T08:19:24.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f3810aca700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f380c15c8a0 0x7f380c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f3810aca700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c0ce740 0x7f380c06a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f3810aca700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c15cc70 con 0x7f380c071220 2023-11-08T08:19:24.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f3810aca700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c06c880 con 0x7f380c15c8a0 2023-11-08T08:19:24.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f3810aca700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c06cad0 con 0x7f380c0ce740 2023-11-08T08:19:24.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f380a59c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f380c15c8a0 0x7f380c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f380a59c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f380c15c8a0 0x7f380c06a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35682/0 (socket says 172.21.15.116:35682) 2023-11-08T08:19:24.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f380a59c700 1 -- 172.21.15.116:0/2495002664 learned_addr learned my addr 172.21.15.116:0/2495002664 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:24.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.950+0000 7f3809d9b700 1 --2- 172.21.15.116:0/2495002664 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c0ce740 0x7f380c06a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380a59c700 1 -- 172.21.15.116:0/2495002664 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c0ce740 msgr2=0x7f380c06a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380a59c700 1 --2- 172.21.15.116:0/2495002664 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c0ce740 0x7f380c06a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380a59c700 1 -- 172.21.15.116:0/2495002664 >> v1:172.21.15.129:6789/0 conn(0x7f380c071220 legacy=0x7f380c0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:24.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380a59c700 1 -- 172.21.15.116:0/2495002664 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c06cdb0 con 0x7f380c15c8a0 2023-11-08T08:19:24.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380a59c700 1 --2- 172.21.15.116:0/2495002664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c15c8a0 0x7f380c06a270 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f38000098f0 tx=0x7f3800009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9da1529d1e167f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380959a700 1 -- 172.21.15.116:0/2495002664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800004230 con 0x7f380c15c8a0 2023-11-08T08:19:24.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380959a700 1 -- 172.21.15.116:0/2495002664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800004390 con 0x7f380c15c8a0 2023-11-08T08:19:24.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f380959a700 1 -- 172.21.15.116:0/2495002664 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000036a0 con 0x7f380c15c8a0 2023-11-08T08:19:24.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f3810aca700 1 -- 172.21.15.116:0/2495002664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c15c8a0 msgr2=0x7f380c06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f3810aca700 1 --2- 172.21.15.116:0/2495002664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c15c8a0 0x7f380c06a270 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f38000098f0 tx=0x7f3800009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:24.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f3810aca700 1 -- 172.21.15.116:0/2495002664 shutdown_connections 2023-11-08T08:19:24.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f3810aca700 1 --2- 172.21.15.116:0/2495002664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c15c8a0 0x7f380c06a270 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f3810aca700 1 --2- 172.21.15.116:0/2495002664 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c0ce740 0x7f380c06a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.951+0000 7f3810aca700 1 -- 172.21.15.116:0/2495002664 >> 172.21.15.116:0/2495002664 conn(0x7f380c06c010 msgr2=0x7f380c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:24.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.952+0000 7f3810aca700 1 -- 172.21.15.116:0/2495002664 shutdown_connections 2023-11-08T08:19:24.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.952+0000 7f3810aca700 1 -- 172.21.15.116:0/2495002664 wait complete. 2023-11-08T08:19:24.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 Processor -- start 2023-11-08T08:19:24.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 -- start start 2023-11-08T08:19:24.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c071220 0x7f380c158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 0x7f380c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f380c15c8a0 0x7f380c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c159190 con 0x7f380c0ce740 2023-11-08T08:19:24.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c159470 con 0x7f380c15c8a0 2023-11-08T08:19:24.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.953+0000 7f3810aca700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c159750 con 0x7f380c071220 2023-11-08T08:19:24.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 0x7f380c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 0x7f380c15b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35684/0 (socket says 172.21.15.116:35684) 2023-11-08T08:19:24.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 -- 172.21.15.116:0/2948805946 learned_addr learned my addr 172.21.15.116:0/2948805946 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:24.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f380a59c700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c071220 0x7f380c158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f380ad9d700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f380c15c8a0 0x7f380c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 -- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c071220 msgr2=0x7f380c158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c071220 0x7f380c158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 -- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f380c15c8a0 msgr2=0x7f380c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:24.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f380c15c8a0 0x7f380c15a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:24.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 -- 172.21.15.116:0/2948805946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800009550 con 0x7f380c0ce740 2023-11-08T08:19:24.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3809d9b700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 0x7f380c15b7d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00e810 tx=0x7f37fc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc00c070 con 0x7f380c0ce740 2023-11-08T08:19:24.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc013610 con 0x7f380c0ce740 2023-11-08T08:19:24.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c06d370 con 0x7f380c0ce740 2023-11-08T08:19:24.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0139c0 con 0x7f380c0ce740 2023-11-08T08:19:24.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.954+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c06d090 con 0x7f380c0ce740 2023-11-08T08:19:24.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.955+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c0cd430 con 0x7f380c0ce740 2023-11-08T08:19:24.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.956+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37fc023070 con 0x7f380c0ce740 2023-11-08T08:19:24.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.956+0000 7f37faffd700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37ec0682f0 0x7f37ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:24.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.956+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f37fc0859e0 con 0x7f380c0ce740 2023-11-08T08:19:24.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.956+0000 7f380a59c700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37ec0682f0 0x7f37ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:24.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.956+0000 7f380a59c700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37ec0682f0 0x7f37ec06a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3800009130 tx=0x7f3800008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:24.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:24.960+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37fc0521c0 con 0x7f380c0ce740 2023-11-08T08:19:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.138+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f380c0743f0 con 0x7f380c0ce740 2023-11-08T08:19:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.138+0000 7f37faffd700 1 -- 172.21.15.116:0/2948805946 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f37fc00f040 con 0x7f380c0ce740 2023-11-08T08:19:25.140 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:25.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37ec0682f0 msgr2=0x7f37ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37ec0682f0 0x7f37ec06a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3800009130 tx=0x7f3800008030 comp rx=0 tx=0).stop 2023-11-08T08:19:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 msgr2=0x7f380c15b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 0x7f380c15b7d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00e810 tx=0x7f37fc00eb20 comp rx=0 tx=0).stop 2023-11-08T08:19:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 shutdown_connections 2023-11-08T08:19:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37ec0682f0 0x7f37ec06a7a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f380c15c8a0 0x7f380c15a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f380c071220 0x7f380c158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 --2- 172.21.15.116:0/2948805946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f380c0ce740 0x7f380c15b7d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 >> 172.21.15.116:0/2948805946 conn(0x7f380c06c010 msgr2=0x7f380c15ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 shutdown_connections 2023-11-08T08:19:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.140+0000 7f3810aca700 1 -- 172.21.15.116:0/2948805946 wait complete. 2023-11-08T08:19:25.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.273+0000 7f66bfa00700 1 Processor -- start 2023-11-08T08:19:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.273+0000 7f66bfa00700 1 -- start start 2023-11-08T08:19:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bfa00700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66b8160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bfa00700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b81607a0 con 0x7f66b80718e0 2023-11-08T08:19:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bfa00700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b8160a50 con 0x7f66b806a270 2023-11-08T08:19:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bfa00700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815cbe0 con 0x7f66b815c810 2023-11-08T08:19:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bcf9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f66b80718e0 0x7f66b806c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56336/0 (socket says 172.21.15.116:56336) 2023-11-08T08:19:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bcf9b700 1 -- 172.21.15.116:0/1707926348 learned_addr learned my addr 172.21.15.116:0/1707926348 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66b8160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608430464 0 0) 0x7f66b81607a0 con 0x7f66b80718e0 2023-11-08T08:19:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a40035b0 con 0x7f66b80718e0 2023-11-08T08:19:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588292285 0 0) 0x7f66b8160a50 con 0x7f66b806a270 2023-11-08T08:19:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b81607a0 con 0x7f66b806a270 2023-11-08T08:19:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4278974917 0 0) 0x7f66a40035b0 con 0x7f66b80718e0 2023-11-08T08:19:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 msgr2=0x7f66b8160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66b8160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 >> v1:172.21.15.129:6789/0 conn(0x7f66b806a270 legacy=0x7f66b80cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b8160d30 con 0x7f66b80718e0 2023-11-08T08:19:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f66a8001670 con 0x7f66b80718e0 2023-11-08T08:19:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 >> v1:172.21.15.116:6789/0 conn(0x7f66b80718e0 legacy=0x7f66b806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4003bc0 0x7f66a4003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66a4004d90 0x7f66a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815cbe0 con 0x7f66a4003bc0 2023-11-08T08:19:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a400bac0 con 0x7f66a4004d90 2023-11-08T08:19:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.274+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a400bc80 con 0x7f66b815c810 2023-11-08T08:19:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4003bc0 0x7f66a4003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bdf9d700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66a4004d90 0x7f66a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bcf9b700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 -- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 msgr2=0x7f66a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 -- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66a4004d90 msgr2=0x7f66a4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66a4004d90 0x7f66a4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 -- 172.21.15.116:0/1707926348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40035b0 con 0x7f66a4003bc0 2023-11-08T08:19:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bdf9d700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66a4004d90 0x7f66a4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bcf9b700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66b815c810 0x7f66a4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4003bc0 0x7f66a4003fb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009b30 tx=0x7f66a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6e0bbfc5d3fd93 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b400e6c0 con 0x7f66a4003bc0 2023-11-08T08:19:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b400e820 con 0x7f66a4003bc0 2023-11-08T08:19:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66affff700 1 -- 172.21.15.116:0/1707926348 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b400eb90 con 0x7f66a4003bc0 2023-11-08T08:19:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bfa00700 1 -- 172.21.15.116:0/1707926348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4003bc0 msgr2=0x7f66a4003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.275+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1707926348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4003bc0 0x7f66a4003fb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009b30 tx=0x7f66a4005880 comp rx=0 tx=0).stop 2023-11-08T08:19:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.276+0000 7f66bdf9d700 1 -- 172.21.15.116:0/1707926348 reap_dead start 2023-11-08T08:19:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.276+0000 7f66bfa00700 1 -- 172.21.15.116:0/1707926348 shutdown_connections 2023-11-08T08:19:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.276+0000 7f66bfa00700 1 -- 172.21.15.116:0/1707926348 >> 172.21.15.116:0/1707926348 conn(0x7f66b806c010 msgr2=0x7f66b80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.276+0000 7f66bfa00700 1 -- 172.21.15.116:0/1707926348 shutdown_connections 2023-11-08T08:19:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.276+0000 7f66bfa00700 1 -- 172.21.15.116:0/1707926348 wait complete. 2023-11-08T08:19:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.277+0000 7f66bfa00700 1 Processor -- start 2023-11-08T08:19:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.277+0000 7f66bfa00700 1 -- start start 2023-11-08T08:19:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bfa00700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66a4003bc0 0x7f66b8158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bfa00700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66b815f080 0x7f66b80cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bfa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 0x7f66b80cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bfa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b81f65b0 con 0x7f66b815f4f0 2023-11-08T08:19:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bfa00700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80c7040 con 0x7f66b815f080 2023-11-08T08:19:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bfa00700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b81586c0 con 0x7f66a4003bc0 2023-11-08T08:19:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bdf9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 0x7f66b80cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bdf9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 0x7f66b80cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35702/0 (socket says 172.21.15.116:35702) 2023-11-08T08:19:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.278+0000 7f66bdf9d700 1 -- 172.21.15.116:0/1953937875 learned_addr learned my addr 172.21.15.116:0/1953937875 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bcf9b700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66b815f080 0x7f66b80cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66a4003bc0 0x7f66b8158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bdf9d700 1 -- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66a4003bc0 msgr2=0x7f66b8158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bdf9d700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66a4003bc0 0x7f66b8158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bdf9d700 1 -- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66b815f080 msgr2=0x7f66b80cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bdf9d700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66b815f080 0x7f66b80cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bdf9d700 1 -- 172.21.15.116:0/1953937875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b80ceb20 con 0x7f66b815f4f0 2023-11-08T08:19:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bcf9b700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66b815f080 0x7f66b80cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bdf9d700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 0x7f66b80cc590 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f66b80d0cd0 tx=0x7f66b000c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0003c10 con 0x7f66b815f4f0 2023-11-08T08:19:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b0003d90 con 0x7f66b815f4f0 2023-11-08T08:19:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b8161010 con 0x7f66b815f4f0 2023-11-08T08:19:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b000c9a0 con 0x7f66b815f4f0 2023-11-08T08:19:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.279+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b806cf90 con 0x7f66b815f4f0 2023-11-08T08:19:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.280+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b80c8830 con 0x7f66b815f4f0 2023-11-08T08:19:25.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.280+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f66b000cb00 con 0x7f66b815f4f0 2023-11-08T08:19:25.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.281+0000 7f66adffb700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f669c0682f0 0x7f669c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.281+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f66b0085f50 con 0x7f66b815f4f0 2023-11-08T08:19:25.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.281+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f669c0682f0 0x7f669c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.281+0000 7f66bd79c700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f669c0682f0 0x7f669c06a7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f66b40090b0 tx=0x7f66b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.285+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66b80c8830 con 0x7f66b815f4f0 2023-11-08T08:19:25.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.480+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f66b80c8830 con 0x7f66b815f4f0 2023-11-08T08:19:25.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.481+0000 7f66adffb700 1 -- 172.21.15.116:0/1953937875 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f66b80c8830 con 0x7f66b815f4f0 2023-11-08T08:19:25.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.482+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f669c0682f0 msgr2=0x7f669c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.482+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f669c0682f0 0x7f669c06a7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f66b40090b0 tx=0x7f66b4008030 comp rx=0 tx=0).stop 2023-11-08T08:19:25.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 msgr2=0x7f66b80cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 0x7f66b80cc590 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f66b80d0cd0 tx=0x7f66b000c0e0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 shutdown_connections 2023-11-08T08:19:25.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b815f4f0 0x7f66b80cc590 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f66a4003bc0 0x7f66b8158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f669c0682f0 0x7f669c06a7a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 --2- 172.21.15.116:0/1953937875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66b815f080 0x7f66b80cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 >> 172.21.15.116:0/1953937875 conn(0x7f66b806c010 msgr2=0x7f66b80caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 shutdown_connections 2023-11-08T08:19:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.483+0000 7f66bfa00700 1 -- 172.21.15.116:0/1953937875 wait complete. 2023-11-08T08:19:25.487 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:25.496 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-08T08:19:25.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:25.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-11-08T08:19:25.529 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:25.530 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:19:25.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:25.530 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-08T08:19:25.588 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:19:25.589 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:25.612 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:19:25.613 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:25.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:25.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.792+0000 7fb220cb6700 1 Processor -- start 2023-11-08T08:19:25.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.792+0000 7fb220cb6700 1 -- start start 2023-11-08T08:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb220cb6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb220cb6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb21c15cc80 0x7fb21c06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb220cb6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c06ab80 con 0x7fb21c06e3a0 2023-11-08T08:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb220cb6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c0b2490 con 0x7fb21c15cc80 2023-11-08T08:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb220cb6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0b1810 con 0x7fb21c06a270 2023-11-08T08:19:25.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb21c15cc80 0x7fb21c06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb21c15cc80 0x7fb21c06de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35704/0 (socket says 172.21.15.116:35704) 2023-11-08T08:19:25.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 -- 172.21.15.116:0/1840093264 learned_addr learned my addr 172.21.15.116:0/1840093264 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:25.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb21a59c700 1 --2- 172.21.15.116:0/1840093264 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 -- 172.21.15.116:0/1840093264 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 msgr2=0x7fb21c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:25.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 --2- 172.21.15.116:0/1840093264 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c06a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 -- 172.21.15.116:0/1840093264 >> v1:172.21.15.129:6789/0 conn(0x7fb21c06e3a0 legacy=0x7fb21c06ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 -- 172.21.15.116:0/1840093264 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0b1af0 con 0x7fb21c15cc80 2023-11-08T08:19:25.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.793+0000 7fb219d9b700 1 --2- 172.21.15.116:0/1840093264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c06de60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb2100098f0 tx=0x7fb210009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bcaa2a87a0afa0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb21959a700 1 -- 172.21.15.116:0/1840093264 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000b8a0 con 0x7fb21c15cc80 2023-11-08T08:19:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb21959a700 1 -- 172.21.15.116:0/1840093264 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21000ba00 con 0x7fb21c15cc80 2023-11-08T08:19:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb21959a700 1 -- 172.21.15.116:0/1840093264 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000bd90 con 0x7fb21c15cc80 2023-11-08T08:19:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 -- 172.21.15.116:0/1840093264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 msgr2=0x7fb21c06de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 --2- 172.21.15.116:0/1840093264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c06de60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb2100098f0 tx=0x7fb210009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:25.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 -- 172.21.15.116:0/1840093264 shutdown_connections 2023-11-08T08:19:25.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 --2- 172.21.15.116:0/1840093264 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c06a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 --2- 172.21.15.116:0/1840093264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c06de60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 -- 172.21.15.116:0/1840093264 >> 172.21.15.116:0/1840093264 conn(0x7fb21c06c010 msgr2=0x7fb21c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:25.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 -- 172.21.15.116:0/1840093264 shutdown_connections 2023-11-08T08:19:25.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.794+0000 7fb220cb6700 1 -- 172.21.15.116:0/1840093264 wait complete. 2023-11-08T08:19:25.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.796+0000 7fb220cb6700 1 Processor -- start 2023-11-08T08:19:25.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.796+0000 7fb220cb6700 1 -- start start 2023-11-08T08:19:25.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb220cb6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb220cb6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb21c06e3a0 0x7fb21c0b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb220cb6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb220cb6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0b4770 con 0x7fb21c15cc80 2023-11-08T08:19:25.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb220cb6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0c9970 con 0x7fb21c06e3a0 2023-11-08T08:19:25.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb220cb6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0b3dd0 con 0x7fb21c06a270 2023-11-08T08:19:25.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c0ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35718/0 (socket says 172.21.15.116:35718) 2023-11-08T08:19:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 -- 172.21.15.116:0/2844112494 learned_addr learned my addr 172.21.15.116:0/2844112494 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21a59c700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb219d9b700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb21c06e3a0 0x7fb21c0b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 -- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 msgr2=0x7fb21c152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 -- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb21c06e3a0 msgr2=0x7fb21c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb21c06e3a0 0x7fb21c0b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 -- 172.21.15.116:0/2844112494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210009550 con 0x7fb21c15cc80 2023-11-08T08:19:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb219d9b700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb21c06e3a0 0x7fb21c0b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb21ad9d700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c0ce8c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb21400cb50 tx=0x7fb21400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214003c00 con 0x7fb21c15cc80 2023-11-08T08:19:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.797+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb214003d60 con 0x7fb21c15cc80 2023-11-08T08:19:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.798+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21400aac0 con 0x7fb21c15cc80 2023-11-08T08:19:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.798+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c0b20b0 con 0x7fb21c15cc80 2023-11-08T08:19:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.798+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c0b1dd0 con 0x7fb21c15cc80 2023-11-08T08:19:25.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.799+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c06e7e0 con 0x7fb21c15cc80 2023-11-08T08:19:25.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.799+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb214010050 con 0x7fb21c15cc80 2023-11-08T08:19:25.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.799+0000 7fb20affd700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb1fc0682f0 0x7fb1fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:25.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.800+0000 7fb21a59c700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb1fc0682f0 0x7fb1fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:25.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.800+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb214085ad0 con 0x7fb21c15cc80 2023-11-08T08:19:25.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.800+0000 7fb21a59c700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb1fc0682f0 0x7fb1fc06a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb20c006b20 tx=0x7fb20c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:25.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.804+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb214052330 con 0x7fb21c15cc80 2023-11-08T08:19:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.984+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb21c0b48b0 con 0x7fb21c15cc80 2023-11-08T08:19:25.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.985+0000 7fb20affd700 1 -- 172.21.15.116:0/2844112494 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7fb214055950 con 0x7fb21c15cc80 2023-11-08T08:19:25.986 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:25.987 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb1fc0682f0 msgr2=0x7fb1fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb1fc0682f0 0x7fb1fc06a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb20c006b20 tx=0x7fb20c008040 comp rx=0 tx=0).stop 2023-11-08T08:19:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 msgr2=0x7fb21c0ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c0ce8c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb21400cb50 tx=0x7fb21400ce60 comp rx=0 tx=0).stop 2023-11-08T08:19:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 shutdown_connections 2023-11-08T08:19:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21c15cc80 0x7fb21c0ce8c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb21c06a270 0x7fb21c152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb1fc0682f0 0x7fb1fc06a7a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 --2- 172.21.15.116:0/2844112494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb21c06e3a0 0x7fb21c0b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 >> 172.21.15.116:0/2844112494 conn(0x7fb21c06c010 msgr2=0x7fb21c0b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 shutdown_connections 2023-11-08T08:19:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:25.987+0000 7fb220cb6700 1 -- 172.21.15.116:0/2844112494 wait complete. 2023-11-08T08:19:26.000 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:26.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.122+0000 7f997b4db700 1 Processor -- start 2023-11-08T08:19:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.122+0000 7f997b4db700 1 -- start start 2023-11-08T08:19:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f997b4db700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f997415c8c0 0x7f9974160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f997b4db700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f997406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f997b4db700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f997406b050 0x7f99740d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f997b4db700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997406b4d0 con 0x7f997406b050 2023-11-08T08:19:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f997b4db700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99740d2d30 con 0x7f997415c8c0 2023-11-08T08:19:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f997b4db700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740d0b40 con 0x7f99740b5dc0 2023-11-08T08:19:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f9979a78700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f997406b050 0x7f99740d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f9979a78700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f997406b050 0x7f99740d0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35732/0 (socket says 172.21.15.116:35732) 2023-11-08T08:19:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.123+0000 7f9979a78700 1 -- 172.21.15.116:0/113783741 learned_addr learned my addr 172.21.15.116:0/113783741 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979a78700 1 -- 172.21.15.116:0/113783741 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 msgr2=0x7f997406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9978a76700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f997406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979277700 1 --2- 172.21.15.116:0/113783741 >> v2:172.21.15.129:3300/0 conn(0x7f997415c8c0 0x7f9974160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979a78700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f997406ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979a78700 1 -- 172.21.15.116:0/113783741 >> v2:172.21.15.129:3300/0 conn(0x7f997415c8c0 msgr2=0x7f9974160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979a78700 1 --2- 172.21.15.116:0/113783741 >> v2:172.21.15.129:3300/0 conn(0x7f997415c8c0 0x7f9974160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979a78700 1 -- 172.21.15.116:0/113783741 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99741611f0 con 0x7f997406b050 2023-11-08T08:19:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9978a76700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f997406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979277700 1 --2- 172.21.15.116:0/113783741 >> v2:172.21.15.129:3300/0 conn(0x7f997415c8c0 0x7f9974160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f9979a78700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997406b050 0x7f99740d0fd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f99740d2ed0 tx=0x7f996c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212c941866abbc43 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f996bfff700 1 -- 172.21.15.116:0/113783741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00e070 con 0x7f997406b050 2023-11-08T08:19:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f996bfff700 1 -- 172.21.15.116:0/113783741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c00b9c0 con 0x7f997406b050 2023-11-08T08:19:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f996bfff700 1 -- 172.21.15.116:0/113783741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00bd70 con 0x7f997406b050 2023-11-08T08:19:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 -- 172.21.15.116:0/113783741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997406b050 msgr2=0x7f99740d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997406b050 0x7f99740d0fd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f99740d2ed0 tx=0x7f996c00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 -- 172.21.15.116:0/113783741 shutdown_connections 2023-11-08T08:19:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997406b050 0x7f99740d0fd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 --2- 172.21.15.116:0/113783741 >> v2:172.21.15.129:3300/0 conn(0x7f997415c8c0 0x7f9974160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 --2- 172.21.15.116:0/113783741 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f997406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.124+0000 7f997b4db700 1 -- 172.21.15.116:0/113783741 >> 172.21.15.116:0/113783741 conn(0x7f997406c520 msgr2=0x7f997406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.125+0000 7f997b4db700 1 -- 172.21.15.116:0/113783741 shutdown_connections 2023-11-08T08:19:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.125+0000 7f997b4db700 1 -- 172.21.15.116:0/113783741 wait complete. 2023-11-08T08:19:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 Processor -- start 2023-11-08T08:19:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 -- start start 2023-11-08T08:19:26.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997406b050 0x7f99740c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f99741f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 0x7f99740b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974149330 con 0x7f997415c8c0 2023-11-08T08:19:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740ca730 con 0x7f997406b050 2023-11-08T08:19:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f997b4db700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740caf20 con 0x7f99740b5dc0 2023-11-08T08:19:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f9979a78700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 0x7f99740b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f9979a78700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 0x7f99740b75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35744/0 (socket says 172.21.15.116:35744) 2023-11-08T08:19:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.126+0000 7f9979a78700 1 -- 172.21.15.116:0/1202776905 learned_addr learned my addr 172.21.15.116:0/1202776905 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979277700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997406b050 0x7f99740c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9978a76700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f99741f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979a78700 1 -- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 msgr2=0x7f99741f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979a78700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f99741f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979a78700 1 -- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997406b050 msgr2=0x7f99740c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979a78700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997406b050 0x7f99740c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979a78700 1 -- 172.21.15.116:0/1202776905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c009b30 con 0x7f997415c8c0 2023-11-08T08:19:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9978a76700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f99741f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9979a78700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 0x7f99740b75f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f99740d0af0 tx=0x7f996c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c004310 con 0x7f997415c8c0 2023-11-08T08:19:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c004470 con 0x7f997415c8c0 2023-11-08T08:19:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c004820 con 0x7f997415c8c0 2023-11-08T08:19:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99740cd760 con 0x7f997415c8c0 2023-11-08T08:19:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.127+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99740d2a50 con 0x7f997415c8c0 2023-11-08T08:19:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.128+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99740cda40 con 0x7f997415c8c0 2023-11-08T08:19:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.129+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f996c02c070 con 0x7f997415c8c0 2023-11-08T08:19:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.129+0000 7f9969ffb700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f99580682f0 0x7f995806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.129+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f996c0873f0 con 0x7f997415c8c0 2023-11-08T08:19:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.129+0000 7f9979277700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f99580682f0 0x7f995806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.129+0000 7f9979277700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f99580682f0 0x7f995806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9970006fd0 tx=0x7f9970008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.133+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f996c053c50 con 0x7f997415c8c0 2023-11-08T08:19:26.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.312+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9974067d70 con 0x7f997415c8c0 2023-11-08T08:19:26.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.313+0000 7f9969ffb700 1 -- 172.21.15.116:0/1202776905 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f996c020020 con 0x7f997415c8c0 2023-11-08T08:19:26.315 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:26.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f99580682f0 msgr2=0x7f995806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f99580682f0 0x7f995806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9970006fd0 tx=0x7f9970008040 comp rx=0 tx=0).stop 2023-11-08T08:19:26.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 msgr2=0x7f99740b75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 0x7f99740b75f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f99740d0af0 tx=0x7f996c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 shutdown_connections 2023-11-08T08:19:26.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f997415c8c0 0x7f99740b75f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f99580682f0 0x7f995806a7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997406b050 0x7f99740c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 --2- 172.21.15.116:0/1202776905 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f99740b5dc0 0x7f99741f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 >> 172.21.15.116:0/1202776905 conn(0x7f997406c520 msgr2=0x7f99740b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:26.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 shutdown_connections 2023-11-08T08:19:26.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.315+0000 7f997b4db700 1 -- 172.21.15.116:0/1202776905 wait complete. 2023-11-08T08:19:26.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.450+0000 7fb2ed2fb700 1 Processor -- start 2023-11-08T08:19:26.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.450+0000 7fb2ed2fb700 1 -- start start 2023-11-08T08:19:26.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2ed2fb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2e80ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2ed2fb700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb2e806a980 0x7fb2e8071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2ed2fb700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e815cc10 con 0x7fb2e8071890 2023-11-08T08:19:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2ed2fb700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e806c9b0 con 0x7fb2e806a980 2023-11-08T08:19:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2ed2fb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e806cc90 con 0x7fb2e80cf710 2023-11-08T08:19:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e67fc700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb2e8071890 0x7fb2e806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56346/0 (socket says 172.21.15.116:56346) 2023-11-08T08:19:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e67fc700 1 -- 172.21.15.116:0/4013314538 learned_addr learned my addr 172.21.15.116:0/4013314538 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2e80ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29190497 0 0) 0x7fb2e815cc10 con 0x7fb2e8071890 2023-11-08T08:19:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e77fe700 1 --2- 172.21.15.116:0/4013314538 >> v2:172.21.15.129:3300/0 conn(0x7fb2e806a980 0x7fb2e8071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d00035b0 con 0x7fb2e8071890 2023-11-08T08:19:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547948240 0 0) 0x7fb2d00035b0 con 0x7fb2e8071890 2023-11-08T08:19:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 msgr2=0x7fb2e80ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2e80ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 >> v2:172.21.15.129:3300/0 conn(0x7fb2e806a980 msgr2=0x7fb2e8071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 --2- 172.21.15.116:0/4013314538 >> v2:172.21.15.129:3300/0 conn(0x7fb2e806a980 0x7fb2e8071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e806cf70 con 0x7fb2e8071890 2023-11-08T08:19:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2e80ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fb2d8001670 con 0x7fb2e8071890 2023-11-08T08:19:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.451+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 >> v1:172.21.15.116:6789/0 conn(0x7fb2e8071890 legacy=0x7fb2e806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2d0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d0004430 0x7fb2d00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2d0005020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e806c9b0 con 0x7fb2d0003a00 2023-11-08T08:19:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e806cc90 con 0x7fb2d0004430 2023-11-08T08:19:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d000ba10 con 0x7fb2e80cf710 2023-11-08T08:19:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2d0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e67fc700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d0004430 0x7fb2d00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e77fe700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2d0005020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 msgr2=0x7fb2d0005020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e80cf710 0x7fb2d0005020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d0004430 msgr2=0x7fb2d00048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d0004430 0x7fb2d00048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/4013314538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0094a0 con 0x7fb2d0003a00 2023-11-08T08:19:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e67fc700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d0004430 0x7fb2d00048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2d0003e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009d30 tx=0x7fb2d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df55584fbadfecb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc009e80 con 0x7fb2d0003a00 2023-11-08T08:19:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2dc00e9a0 con 0x7fb2d0003a00 2023-11-08T08:19:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.452+0000 7fb2e5ffb700 1 -- 172.21.15.116:0/4013314538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc00ed10 con 0x7fb2d0003a00 2023-11-08T08:19:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/4013314538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 msgr2=0x7fb2d0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/4013314538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2d0003e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009d30 tx=0x7fb2d0005880 comp rx=0 tx=0).stop 2023-11-08T08:19:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2e77fe700 1 -- 172.21.15.116:0/4013314538 reap_dead start 2023-11-08T08:19:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/4013314538 shutdown_connections 2023-11-08T08:19:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/4013314538 >> 172.21.15.116:0/4013314538 conn(0x7fb2e806c010 msgr2=0x7fb2e80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/4013314538 shutdown_connections 2023-11-08T08:19:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.453+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/4013314538 wait complete. 2023-11-08T08:19:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.454+0000 7fb2ed2fb700 1 Processor -- start 2023-11-08T08:19:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.454+0000 7fb2ed2fb700 1 -- start start 2023-11-08T08:19:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2e8151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e81f5630 0x7fb2e81f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e81e62f0 0x7fb2e81e6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e8151c80 con 0x7fb2d0003a00 2023-11-08T08:19:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81e6ca0 con 0x7fb2e81f5630 2023-11-08T08:19:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80b3c40 con 0x7fb2e81e62f0 2023-11-08T08:19:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2e8151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2e8151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35766/0 (socket says 172.21.15.116:35766) 2023-11-08T08:19:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/3147815395 learned_addr learned my addr 172.21.15.116:0/3147815395 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e67fc700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e81f5630 0x7fb2e81f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e77fe700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e81e62f0 0x7fb2e81e6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e81e62f0 msgr2=0x7fb2e81e6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e81e62f0 0x7fb2e81e6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e81f5630 msgr2=0x7fb2e81f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e81f5630 0x7fb2e81f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 -- 172.21.15.116:0/3147815395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e806d530 con 0x7fb2d0003a00 2023-11-08T08:19:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e67fc700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e81f5630 0x7fb2e81f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2e6ffd700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2e8151740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00e790 tx=0x7fb2dc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc003ab0 con 0x7fb2d0003a00 2023-11-08T08:19:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2dc003c10 con 0x7fb2d0003a00 2023-11-08T08:19:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e806d250 con 0x7fb2d0003a00 2023-11-08T08:19:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc009e80 con 0x7fb2d0003a00 2023-11-08T08:19:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.455+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e80ce740 con 0x7fb2d0003a00 2023-11-08T08:19:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.456+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e80743f0 con 0x7fb2d0003a00 2023-11-08T08:19:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.457+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2dc019030 con 0x7fb2d0003a00 2023-11-08T08:19:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.457+0000 7fb2cf7fe700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb2d40682f0 0x7fb2d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.457+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb2dc087ea0 con 0x7fb2d0003a00 2023-11-08T08:19:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.457+0000 7fb2e67fc700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb2d40682f0 0x7fb2d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.458+0000 7fb2e67fc700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb2d40682f0 0x7fb2d406a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8001eb0 tx=0x7fb2d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.461+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2dc0580c0 con 0x7fb2d0003a00 2023-11-08T08:19:26.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.641+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2e81f6180 con 0x7fb2d0003a00 2023-11-08T08:19:26.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.642+0000 7fb2cf7fe700 1 -- 172.21.15.116:0/3147815395 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2dc035020 con 0x7fb2d0003a00 2023-11-08T08:19:26.643 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:26.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb2d40682f0 msgr2=0x7fb2d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb2d40682f0 0x7fb2d406a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8001eb0 tx=0x7fb2d8009040 comp rx=0 tx=0).stop 2023-11-08T08:19:26.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 msgr2=0x7fb2e8151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2e8151740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00e790 tx=0x7fb2dc0047c0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 shutdown_connections 2023-11-08T08:19:26.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb2e81e62f0 0x7fb2e81e6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2d0003a00 0x7fb2e8151740 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb2d40682f0 0x7fb2d406a7a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 --2- 172.21.15.116:0/3147815395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e81f5630 0x7fb2e81f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 >> 172.21.15.116:0/3147815395 conn(0x7fb2e806c010 msgr2=0x7fb2e80b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:26.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 shutdown_connections 2023-11-08T08:19:26.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.644+0000 7fb2ed2fb700 1 -- 172.21.15.116:0/3147815395 wait complete. 2023-11-08T08:19:26.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:26.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.779+0000 7f9366c9d700 1 Processor -- start 2023-11-08T08:19:26.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.779+0000 7f9366c9d700 1 -- start start 2023-11-08T08:19:26.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f9366c9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9360158e10 0x7f93600cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f9366c9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93600ae850 0x7f936006c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f9366c9d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93600ca140 con 0x7f936006c980 2023-11-08T08:19:26.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f9366c9d700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936006ac30 con 0x7f93600ae850 2023-11-08T08:19:26.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f9366c9d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936006ce00 con 0x7f9360158e10 2023-11-08T08:19:26.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93600ae850 0x7f936006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93600ae850 0x7f936006c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35778/0 (socket says 172.21.15.116:35778) 2023-11-08T08:19:26.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 -- 172.21.15.116:0/2909928553 learned_addr learned my addr 172.21.15.116:0/2909928553 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:26.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f9364a39700 1 --2- 172.21.15.116:0/2909928553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9360158e10 0x7f93600cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 -- 172.21.15.116:0/2909928553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9360158e10 msgr2=0x7f93600cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:26.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 --2- 172.21.15.116:0/2909928553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9360158e10 0x7f93600cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 -- 172.21.15.116:0/2909928553 >> v1:172.21.15.129:6789/0 conn(0x7f936006c980 legacy=0x7f936015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 -- 172.21.15.116:0/2909928553 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360071380 con 0x7f93600ae850 2023-11-08T08:19:26.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935ffff700 1 --2- 172.21.15.116:0/2909928553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93600ae850 0x7f936006c440 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f934c00bb90 tx=0x7f934c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ae0da2154b97c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935f7fe700 1 -- 172.21.15.116:0/2909928553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c00dd50 con 0x7f93600ae850 2023-11-08T08:19:26.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.780+0000 7f935f7fe700 1 -- 172.21.15.116:0/2909928553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c010470 con 0x7f93600ae850 2023-11-08T08:19:26.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f935f7fe700 1 -- 172.21.15.116:0/2909928553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c010800 con 0x7f93600ae850 2023-11-08T08:19:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 -- 172.21.15.116:0/2909928553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93600ae850 msgr2=0x7f936006c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 --2- 172.21.15.116:0/2909928553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93600ae850 0x7f936006c440 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f934c00bb90 tx=0x7f934c00bea0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 -- 172.21.15.116:0/2909928553 shutdown_connections 2023-11-08T08:19:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 --2- 172.21.15.116:0/2909928553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9360158e10 0x7f93600cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 --2- 172.21.15.116:0/2909928553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93600ae850 0x7f936006c440 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 -- 172.21.15.116:0/2909928553 >> 172.21.15.116:0/2909928553 conn(0x7f93600cf400 msgr2=0x7f93600cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 -- 172.21.15.116:0/2909928553 shutdown_connections 2023-11-08T08:19:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.781+0000 7f9366c9d700 1 -- 172.21.15.116:0/2909928553 wait complete. 2023-11-08T08:19:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.782+0000 7f9366c9d700 1 Processor -- start 2023-11-08T08:19:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.782+0000 7f9366c9d700 1 -- start start 2023-11-08T08:19:26.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9366c9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f936006c980 0x7f93600c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9366c9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93600ae850 0x7f93600c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9366c9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 0x7f93600ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9366c9d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93600b43d0 con 0x7f9360158e10 2023-11-08T08:19:26.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9366c9d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93600c46f0 con 0x7f936006c980 2023-11-08T08:19:26.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9366c9d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93600c5c40 con 0x7f93600ae850 2023-11-08T08:19:26.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 0x7f93600ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 0x7f93600ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35780/0 (socket says 172.21.15.116:35780) 2023-11-08T08:19:26.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 -- 172.21.15.116:0/3885333173 learned_addr learned my addr 172.21.15.116:0/3885333173 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:26.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f935ffff700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93600ae850 0x7f93600c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f9364a39700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f936006c980 0x7f93600c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 -- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93600ae850 msgr2=0x7f93600c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93600ae850 0x7f93600c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 -- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f936006c980 msgr2=0x7f93600c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f936006c980 0x7f93600c55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.783+0000 7f936523a700 1 -- 172.21.15.116:0/3885333173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c00b7f0 con 0x7f9360158e10 2023-11-08T08:19:26.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.784+0000 7f936523a700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 0x7f93600ad490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f935800cc50 tx=0x7f935800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.784+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935800ad80 con 0x7f9360158e10 2023-11-08T08:19:26.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.784+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935800ce50 con 0x7f9360158e10 2023-11-08T08:19:26.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.784+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93600c9e60 con 0x7f9360158e10 2023-11-08T08:19:26.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.784+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358003c40 con 0x7f9360158e10 2023-11-08T08:19:26.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.784+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93600c9b80 con 0x7f9360158e10 2023-11-08T08:19:26.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.785+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936006ae20 con 0x7f9360158e10 2023-11-08T08:19:26.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.785+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9358010050 con 0x7f9360158e10 2023-11-08T08:19:26.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.786+0000 7f935d7fa700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93440682f0 0x7f934406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:26.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.786+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f93580866f0 con 0x7f9360158e10 2023-11-08T08:19:26.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.786+0000 7f9364a39700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93440682f0 0x7f934406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:26.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.786+0000 7f9364a39700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93440682f0 0x7f934406a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93600ae7c0 tx=0x7f9354008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:26.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.789+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9358052ff0 con 0x7f9360158e10 2023-11-08T08:19:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.968+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93600b4510 con 0x7f9360158e10 2023-11-08T08:19:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.968+0000 7f935d7fa700 1 -- 172.21.15.116:0/3885333173 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f93600b4510 con 0x7f9360158e10 2023-11-08T08:19:26.970 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:26.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93440682f0 msgr2=0x7f934406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93440682f0 0x7f934406a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93600ae7c0 tx=0x7f9354008040 comp rx=0 tx=0).stop 2023-11-08T08:19:26.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 msgr2=0x7f93600ad490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:26.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 0x7f93600ad490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f935800cc50 tx=0x7f935800a340 comp rx=0 tx=0).stop 2023-11-08T08:19:26.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 shutdown_connections 2023-11-08T08:19:26.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360158e10 0x7f93600ad490 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93440682f0 0x7f934406a7a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f936006c980 0x7f93600c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 --2- 172.21.15.116:0/3885333173 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93600ae850 0x7f93600c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:26.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 >> 172.21.15.116:0/3885333173 conn(0x7f93600cf400 msgr2=0x7f93600af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:26.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 shutdown_connections 2023-11-08T08:19:26.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:26.970+0000 7f9366c9d700 1 -- 172.21.15.116:0/3885333173 wait complete. 2023-11-08T08:19:26.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:27.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 Processor -- start 2023-11-08T08:19:27.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 -- start start 2023-11-08T08:19:27.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57c0160300 0x7f57c015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f57c015d880 0x7f57c00cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c015ceb0 con 0x7f57c00cec00 2023-11-08T08:19:27.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c01608c0 con 0x7f57c0160300 2023-11-08T08:19:27.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.105+0000 7f57c67d3700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c0160ba0 con 0x7f57c015d880 2023-11-08T08:19:27.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57c0160300 0x7f57c015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57c0160300 0x7f57c015d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35792/0 (socket says 172.21.15.116:35792) 2023-11-08T08:19:27.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 -- 172.21.15.116:0/706350063 learned_addr learned my addr 172.21.15.116:0/706350063 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:27.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57c4d70700 1 --2- 172.21.15.116:0/706350063 >> v2:172.21.15.129:3300/0 conn(0x7f57c015d880 0x7f57c00cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 -- 172.21.15.116:0/706350063 >> v2:172.21.15.129:3300/0 conn(0x7f57c015d880 msgr2=0x7f57c00cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 --2- 172.21.15.116:0/706350063 >> v2:172.21.15.129:3300/0 conn(0x7f57c015d880 0x7f57c00cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 -- 172.21.15.116:0/706350063 >> v1:172.21.15.129:6789/0 conn(0x7f57c00cec00 legacy=0x7f57c0065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 -- 172.21.15.116:0/706350063 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0160e80 con 0x7f57c0160300 2023-11-08T08:19:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57bf7fe700 1 --2- 172.21.15.116:0/706350063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0160300 0x7f57c015d340 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f57c006bae0 tx=0x7f57b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85707b6421aa2b53 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57beffd700 1 -- 172.21.15.116:0/706350063 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4004230 con 0x7f57c0160300 2023-11-08T08:19:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57beffd700 1 -- 172.21.15.116:0/706350063 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4004390 con 0x7f57c0160300 2023-11-08T08:19:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.106+0000 7f57beffd700 1 -- 172.21.15.116:0/706350063 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b40036a0 con 0x7f57c0160300 2023-11-08T08:19:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 -- 172.21.15.116:0/706350063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0160300 msgr2=0x7f57c015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 --2- 172.21.15.116:0/706350063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0160300 0x7f57c015d340 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f57c006bae0 tx=0x7f57b4009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 -- 172.21.15.116:0/706350063 shutdown_connections 2023-11-08T08:19:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 --2- 172.21.15.116:0/706350063 >> v2:172.21.15.129:3300/0 conn(0x7f57c015d880 0x7f57c00cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 --2- 172.21.15.116:0/706350063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0160300 0x7f57c015d340 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 -- 172.21.15.116:0/706350063 >> 172.21.15.116:0/706350063 conn(0x7f57c00cf400 msgr2=0x7f57c00cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 -- 172.21.15.116:0/706350063 shutdown_connections 2023-11-08T08:19:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.107+0000 7f57c67d3700 1 -- 172.21.15.116:0/706350063 wait complete. 2023-11-08T08:19:27.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.108+0000 7f57c67d3700 1 Processor -- start 2023-11-08T08:19:27.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.108+0000 7f57c67d3700 1 -- start start 2023-11-08T08:19:27.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57c67d3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57c00cec00 0x7f57c00b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57c67d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 0x7f57c00b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57c67d3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57c0160300 0x7f57c0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57c67d3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0156f80 con 0x7f57c015d880 2023-11-08T08:19:27.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57c67d3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00c7010 con 0x7f57c0160300 2023-11-08T08:19:27.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57c67d3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00c6670 con 0x7f57c00cec00 2023-11-08T08:19:27.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57bf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 0x7f57c00b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57bf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 0x7f57c00b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35806/0 (socket says 172.21.15.116:35806) 2023-11-08T08:19:27.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.109+0000 7f57bf7fe700 1 -- 172.21.15.116:0/338379875 learned_addr learned my addr 172.21.15.116:0/338379875 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:27.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57c4d70700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57c0160300 0x7f57c0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bffff700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57c00cec00 0x7f57c00b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bf7fe700 1 -- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57c00cec00 msgr2=0x7f57c00b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bf7fe700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57c00cec00 0x7f57c00b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bf7fe700 1 -- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57c0160300 msgr2=0x7f57c0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bf7fe700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57c0160300 0x7f57c0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bf7fe700 1 -- 172.21.15.116:0/338379875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b4009550 con 0x7f57c015d880 2023-11-08T08:19:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bffff700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57c00cec00 0x7f57c00b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57c4d70700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57c0160300 0x7f57c0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bf7fe700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 0x7f57c00b4170 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009130 tx=0x7f57b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4011630 con 0x7f57c015d880 2023-11-08T08:19:27.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4011790 con 0x7f57c015d880 2023-11-08T08:19:27.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c00cefd0 con 0x7f57c015d880 2023-11-08T08:19:27.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4011b40 con 0x7f57c015d880 2023-11-08T08:19:27.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.110+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c0161160 con 0x7f57c015d880 2023-11-08T08:19:27.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.111+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c00cf160 con 0x7f57c015d880 2023-11-08T08:19:27.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.111+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57b4017020 con 0x7f57c015d880 2023-11-08T08:19:27.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.112+0000 7f57bcff9700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57ac0682f0 0x7f57ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.112+0000 7f57bffff700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57ac0682f0 0x7f57ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.112+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f57b40872a0 con 0x7f57c015d880 2023-11-08T08:19:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.112+0000 7f57bffff700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57ac0682f0 0x7f57ac06a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f57b0001750 tx=0x7f57b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.116+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57b4053b00 con 0x7f57c015d880 2023-11-08T08:19:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.294+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f57c00c67b0 con 0x7f57c015d880 2023-11-08T08:19:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.295+0000 7f57bcff9700 1 -- 172.21.15.116:0/338379875 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57c00c67b0 con 0x7f57c015d880 2023-11-08T08:19:27.296 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57ac0682f0 msgr2=0x7f57ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57ac0682f0 0x7f57ac06a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f57b0001750 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2023-11-08T08:19:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 msgr2=0x7f57c00b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 0x7f57c00b4170 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009130 tx=0x7f57b4004750 comp rx=0 tx=0).stop 2023-11-08T08:19:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 shutdown_connections 2023-11-08T08:19:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57ac0682f0 0x7f57ac06a7a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57c0160300 0x7f57c0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57c00cec00 0x7f57c00b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 --2- 172.21.15.116:0/338379875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c015d880 0x7f57c00b4170 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 >> 172.21.15.116:0/338379875 conn(0x7f57c00cf400 msgr2=0x7f57c015e430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 shutdown_connections 2023-11-08T08:19:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.297+0000 7f57c67d3700 1 -- 172.21.15.116:0/338379875 wait complete. 2023-11-08T08:19:27.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.429+0000 7fd076bf8700 1 Processor -- start 2023-11-08T08:19:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.430+0000 7fd076bf8700 1 -- start start 2023-11-08T08:19:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.430+0000 7fd076bf8700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd07015c810 0x7fd070160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.430+0000 7fd076bf8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 0x7fd07006c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.430+0000 7fd076bf8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0701607a0 con 0x7fd07006cdc0 2023-11-08T08:19:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.430+0000 7fd076bf8700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07015cbe0 con 0x7fd07015c810 2023-11-08T08:19:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.430+0000 7fd076bf8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd070160a50 con 0x7fd0700718e0 2023-11-08T08:19:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd074994700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd07015c810 0x7fd070160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 0x7fd07006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 0x7fd07006c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:44700/0 (socket says 172.21.15.116:44700) 2023-11-08T08:19:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 -- 172.21.15.116:0/1730597314 learned_addr learned my addr 172.21.15.116:0/1730597314 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 -- 172.21.15.116:0/1730597314 >> v2:172.21.15.129:3300/0 conn(0x7fd07015c810 msgr2=0x7fd070160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 --2- 172.21.15.116:0/1730597314 >> v2:172.21.15.129:3300/0 conn(0x7fd07015c810 0x7fd070160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 -- 172.21.15.116:0/1730597314 >> v1:172.21.15.129:6789/0 conn(0x7fd07006cdc0 legacy=0x7fd07006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 -- 172.21.15.116:0/1730597314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070160d30 con 0x7fd0700718e0 2023-11-08T08:19:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06ffff700 1 --2- 172.21.15.116:0/1730597314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 0x7fd07006c880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0098f0 tx=0x7fd05c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ccbeff18ebd45ae server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.431+0000 7fd06f7fe700 1 -- 172.21.15.116:0/1730597314 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c011830 con 0x7fd0700718e0 2023-11-08T08:19:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd06f7fe700 1 -- 172.21.15.116:0/1730597314 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c011990 con 0x7fd0700718e0 2023-11-08T08:19:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd06f7fe700 1 -- 172.21.15.116:0/1730597314 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c011d20 con 0x7fd0700718e0 2023-11-08T08:19:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 -- 172.21.15.116:0/1730597314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 msgr2=0x7fd07006c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 --2- 172.21.15.116:0/1730597314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 0x7fd07006c880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0098f0 tx=0x7fd05c009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 -- 172.21.15.116:0/1730597314 shutdown_connections 2023-11-08T08:19:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 --2- 172.21.15.116:0/1730597314 >> v2:172.21.15.129:3300/0 conn(0x7fd07015c810 0x7fd070160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 --2- 172.21.15.116:0/1730597314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0700718e0 0x7fd07006c880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 -- 172.21.15.116:0/1730597314 >> 172.21.15.116:0/1730597314 conn(0x7fd07006c010 msgr2=0x7fd0700cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 -- 172.21.15.116:0/1730597314 shutdown_connections 2023-11-08T08:19:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.432+0000 7fd076bf8700 1 -- 172.21.15.116:0/1730597314 wait complete. 2023-11-08T08:19:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.433+0000 7fd076bf8700 1 Processor -- start 2023-11-08T08:19:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.433+0000 7fd076bf8700 1 -- start start 2023-11-08T08:19:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd076bf8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd07006cdc0 0x7fd070159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd076bf8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0700718e0 0x7fd0700c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd076bf8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 0x7fd07014f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd076bf8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07015da80 con 0x7fd07015c810 2023-11-08T08:19:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd076bf8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07015ac80 con 0x7fd0700718e0 2023-11-08T08:19:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd076bf8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07015b8e0 con 0x7fd07006cdc0 2023-11-08T08:19:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd075195700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 0x7fd07014f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd075195700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 0x7fd07014f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35810/0 (socket says 172.21.15.116:35810) 2023-11-08T08:19:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd075195700 1 -- 172.21.15.116:0/3271945704 learned_addr learned my addr 172.21.15.116:0/3271945704 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd06ffff700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0700718e0 0x7fd0700c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.434+0000 7fd074994700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd07006cdc0 0x7fd070159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd075195700 1 -- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd07006cdc0 msgr2=0x7fd070159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd075195700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd07006cdc0 0x7fd070159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd075195700 1 -- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0700718e0 msgr2=0x7fd0700c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd075195700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0700718e0 0x7fd0700c5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd075195700 1 -- 172.21.15.116:0/3271945704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c009550 con 0x7fd07015c810 2023-11-08T08:19:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd075195700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 0x7fd07014f0f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd06800caf0 tx=0x7fd06800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068003c00 con 0x7fd07015c810 2023-11-08T08:19:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd068003d60 con 0x7fd07015c810 2023-11-08T08:19:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0700ceb20 con 0x7fd07015c810 2023-11-08T08:19:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06800ab10 con 0x7fd07015c810 2023-11-08T08:19:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.435+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd070161010 con 0x7fd07015c810 2023-11-08T08:19:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.436+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0700ce740 con 0x7fd07015c810 2023-11-08T08:19:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.436+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd068010050 con 0x7fd07015c810 2023-11-08T08:19:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.436+0000 7fd06d7fa700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0540682f0 0x7fd05406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.437+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd068085bf0 con 0x7fd07015c810 2023-11-08T08:19:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.437+0000 7fd074994700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0540682f0 0x7fd05406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.437+0000 7fd074994700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0540682f0 0x7fd05406a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd064009b70 tx=0x7fd064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.441+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0680523d0 con 0x7fd07015c810 2023-11-08T08:19:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.636+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd0700743f0 con 0x7fd07015c810 2023-11-08T08:19:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.637+0000 7fd06d7fa700 1 -- 172.21.15.116:0/3271945704 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd0680559f0 con 0x7fd07015c810 2023-11-08T08:19:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.638+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0540682f0 msgr2=0x7fd05406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.638+0000 7fd076bf8700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0540682f0 0x7fd05406a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd064009b70 tx=0x7fd064008040 comp rx=0 tx=0).stop 2023-11-08T08:19:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.638+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 msgr2=0x7fd07014f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.638+0000 7fd076bf8700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 0x7fd07014f0f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd06800caf0 tx=0x7fd06800ceb0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 shutdown_connections 2023-11-08T08:19:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07015c810 0x7fd07014f0f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd07006cdc0 0x7fd070159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0540682f0 0x7fd05406a7a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 --2- 172.21.15.116:0/3271945704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0700718e0 0x7fd0700c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 >> 172.21.15.116:0/3271945704 conn(0x7fd07006c010 msgr2=0x7fd0700cc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 shutdown_connections 2023-11-08T08:19:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.639+0000 7fd076bf8700 1 -- 172.21.15.116:0/3271945704 wait complete. 2023-11-08T08:19:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:27.652 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:19:27.652 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:27.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.795+0000 7f35d6a71700 1 Processor -- start 2023-11-08T08:19:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.795+0000 7f35d6a71700 1 -- start start 2023-11-08T08:19:27.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d6a71700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f35d0071f70 0x7f35d00cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d6a71700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35d00b4a30 0x7f35d00b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d6a71700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d00cfb00 con 0x7f35d00b4660 2023-11-08T08:19:27.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d6a71700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d015aa20 con 0x7f35d00b4a30 2023-11-08T08:19:27.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d6a71700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d00cf4d0 con 0x7f35d0071f70 2023-11-08T08:19:27.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35d00b4a30 0x7f35d00b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35d00b4a30 0x7f35d00b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35822/0 (socket says 172.21.15.116:35822) 2023-11-08T08:19:27.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 -- 172.21.15.116:0/175067742 learned_addr learned my addr 172.21.15.116:0/175067742 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:27.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35cffff700 1 --2- 172.21.15.116:0/175067742 >> v2:172.21.15.129:3300/0 conn(0x7f35d0071f70 0x7f35d00cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 -- 172.21.15.116:0/175067742 >> v2:172.21.15.129:3300/0 conn(0x7f35d0071f70 msgr2=0x7f35d00cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:27.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 --2- 172.21.15.116:0/175067742 >> v2:172.21.15.129:3300/0 conn(0x7f35d0071f70 0x7f35d00cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 -- 172.21.15.116:0/175067742 >> v1:172.21.15.129:6789/0 conn(0x7f35d00b4660 legacy=0x7f35d006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 -- 172.21.15.116:0/175067742 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0072550 con 0x7f35d00b4a30 2023-11-08T08:19:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35cffff700 1 --2- 172.21.15.116:0/175067742 >> v2:172.21.15.129:3300/0 conn(0x7f35d0071f70 0x7f35d00cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35d500e700 1 --2- 172.21.15.116:0/175067742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d00b4e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f35d00c9870 tx=0x7f35c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2aa0d0c8a03fab server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35cf7fe700 1 -- 172.21.15.116:0/175067742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c800e070 con 0x7f35d00b4a30 2023-11-08T08:19:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35cf7fe700 1 -- 172.21.15.116:0/175067742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c8003a20 con 0x7f35d00b4a30 2023-11-08T08:19:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.796+0000 7f35cf7fe700 1 -- 172.21.15.116:0/175067742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8003dd0 con 0x7f35d00b4a30 2023-11-08T08:19:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 -- 172.21.15.116:0/175067742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 msgr2=0x7f35d00b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 --2- 172.21.15.116:0/175067742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d00b4e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f35d00c9870 tx=0x7f35c800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 -- 172.21.15.116:0/175067742 shutdown_connections 2023-11-08T08:19:27.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 --2- 172.21.15.116:0/175067742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d00b4e90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 --2- 172.21.15.116:0/175067742 >> v2:172.21.15.129:3300/0 conn(0x7f35d0071f70 0x7f35d00cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 -- 172.21.15.116:0/175067742 >> 172.21.15.116:0/175067742 conn(0x7f35d00b2fc0 msgr2=0x7f35d0071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:27.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 -- 172.21.15.116:0/175067742 shutdown_connections 2023-11-08T08:19:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.797+0000 7f35d6a71700 1 -- 172.21.15.116:0/175067742 wait complete. 2023-11-08T08:19:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.798+0000 7f35d6a71700 1 Processor -- start 2023-11-08T08:19:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.798+0000 7f35d6a71700 1 -- start start 2023-11-08T08:19:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d6a71700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35d0071f70 0x7f35d0155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d6a71700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35d00b4660 0x7f35d015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d6a71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d015c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d6a71700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00c6620 con 0x7f35d00b4a30 2023-11-08T08:19:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d6a71700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d015c9e0 con 0x7f35d0071f70 2023-11-08T08:19:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d6a71700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d015ccc0 con 0x7f35d00b4660 2023-11-08T08:19:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d015c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d015c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35838/0 (socket says 172.21.15.116:35838) 2023-11-08T08:19:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 -- 172.21.15.116:0/2193423568 learned_addr learned my addr 172.21.15.116:0/2193423568 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35cffff700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35d00b4660 0x7f35d015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 -- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35d00b4660 msgr2=0x7f35d015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35d00b4660 0x7f35d015a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 -- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35d0071f70 msgr2=0x7f35d0155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d480d700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35d0071f70 0x7f35d0155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35d0071f70 0x7f35d0155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 -- 172.21.15.116:0/2193423568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8009b30 con 0x7f35d00b4a30 2023-11-08T08:19:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d480d700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35d0071f70 0x7f35d0155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.799+0000 7f35d500e700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d015c2e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f35c8015040 tx=0x7f35c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.800+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8004240 con 0x7f35d00b4a30 2023-11-08T08:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.800+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c80043c0 con 0x7f35d00b4a30 2023-11-08T08:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.800+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8004770 con 0x7f35d00b4a30 2023-11-08T08:19:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.800+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d015f170 con 0x7f35d00b4a30 2023-11-08T08:19:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.800+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00cf790 con 0x7f35d00b4a30 2023-11-08T08:19:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.801+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d015f300 con 0x7f35d00b4a30 2023-11-08T08:19:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.801+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f35c8012430 con 0x7f35d00b4a30 2023-11-08T08:19:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.801+0000 7f35cd7fa700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35b40682f0 0x7f35b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.801+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f35c8087030 con 0x7f35d00b4a30 2023-11-08T08:19:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.801+0000 7f35d480d700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35b40682f0 0x7f35b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.802+0000 7f35d480d700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35b40682f0 0x7f35b406a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35c4003f40 tx=0x7f35c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.806+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35c80537b0 con 0x7f35d00b4a30 2023-11-08T08:19:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.986+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35d015cf00 con 0x7f35d00b4a30 2023-11-08T08:19:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.987+0000 7f35cd7fa700 1 -- 172.21.15.116:0/2193423568 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f35d015cf00 con 0x7f35d00b4a30 2023-11-08T08:19:27.988 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:27.989 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35b40682f0 msgr2=0x7f35b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35b40682f0 0x7f35b406a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35c4003f40 tx=0x7f35c4009040 comp rx=0 tx=0).stop 2023-11-08T08:19:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 msgr2=0x7f35d015c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d015c2e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f35c8015040 tx=0x7f35c8004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 shutdown_connections 2023-11-08T08:19:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d00b4a30 0x7f35d015c2e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35b40682f0 0x7f35b406a7a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35d0071f70 0x7f35d0155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 --2- 172.21.15.116:0/2193423568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35d00b4660 0x7f35d015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 >> 172.21.15.116:0/2193423568 conn(0x7f35d00b2fc0 msgr2=0x7f35d006c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 shutdown_connections 2023-11-08T08:19:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:27.989+0000 7f35d6a71700 1 -- 172.21.15.116:0/2193423568 wait complete. 2023-11-08T08:19:28.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:28.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.125+0000 7f4f6c5dd700 1 Processor -- start 2023-11-08T08:19:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.125+0000 7f4f6c5dd700 1 -- start start 2023-11-08T08:19:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6c5dd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f641599b0 0x7f4f6415d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6c5dd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 0x7f4f6415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6c5dd700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f64159ec0 con 0x7f4f640b1650 2023-11-08T08:19:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6c5dd700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6415d8a0 con 0x7f4f641599b0 2023-11-08T08:19:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6c5dd700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6415db80 con 0x7f4f640af0a0 2023-11-08T08:19:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6a379700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4f640b1650 0x7f4f640b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56348/0 (socket says 172.21.15.116:56348) 2023-11-08T08:19:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6a379700 1 -- 172.21.15.116:0/756570972 learned_addr learned my addr 172.21.15.116:0/756570972 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69b78700 1 --2- 172.21.15.116:0/756570972 >> v2:172.21.15.129:3300/0 conn(0x7f4f641599b0 0x7f4f6415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f6ab7a700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 0x7f4f6415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801432499 0 0) 0x7f4f64159ec0 con 0x7f4f640b1650 2023-11-08T08:19:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500035b0 con 0x7f4f640b1650 2023-11-08T08:19:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744789477 0 0) 0x7f4f500035b0 con 0x7f4f640b1650 2023-11-08T08:19:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 msgr2=0x7f4f6415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 0x7f4f6415a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 >> v2:172.21.15.129:3300/0 conn(0x7f4f641599b0 msgr2=0x7f4f6415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 --2- 172.21.15.116:0/756570972 >> v2:172.21.15.129:3300/0 conn(0x7f4f641599b0 0x7f4f6415d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6415de60 con 0x7f4f640b1650 2023-11-08T08:19:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.126+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f4f54001670 con 0x7f4f640b1650 2023-11-08T08:19:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 >> v1:172.21.15.116:6789/0 conn(0x7f4f640b1650 legacy=0x7f4f640b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50003a00 0x7f4f50003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 0x7f4f50004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6415d8a0 con 0x7f4f50003a00 2023-11-08T08:19:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6415db80 con 0x7f4f50004d90 2023-11-08T08:19:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000ba10 con 0x7f4f640af0a0 2023-11-08T08:19:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50003a00 0x7f4f50003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69b78700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 0x7f4f50004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6ab7a700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 -- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 msgr2=0x7f4f50004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f640af0a0 0x7f4f50004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 -- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f50004d90 msgr2=0x7f4f50005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 -- 172.21.15.116:0/756570972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f500035b0 con 0x7f4f50003a00 2023-11-08T08:19:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6ab7a700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f6a379700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50003a00 0x7f4f50003e10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f54007f60 tx=0x7f4f50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1816feb0bb364ca server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5400cda0 con 0x7f4f50003a00 2023-11-08T08:19:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f54005960 con 0x7f4f50003a00 2023-11-08T08:19:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.127+0000 7f4f69377700 1 -- 172.21.15.116:0/756570972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54005d10 con 0x7f4f50003a00 2023-11-08T08:19:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/756570972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50003a00 msgr2=0x7f4f50003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/756570972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50003a00 0x7f4f50003e10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f54007f60 tx=0x7f4f50005880 comp rx=0 tx=0).stop 2023-11-08T08:19:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6ab7a700 1 -- 172.21.15.116:0/756570972 reap_dead start 2023-11-08T08:19:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/756570972 shutdown_connections 2023-11-08T08:19:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/756570972 >> 172.21.15.116:0/756570972 conn(0x7f4f6406c010 msgr2=0x7f4f640cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/756570972 shutdown_connections 2023-11-08T08:19:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.128+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/756570972 wait complete. 2023-11-08T08:19:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.129+0000 7f4f6c5dd700 1 Processor -- start 2023-11-08T08:19:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.129+0000 7f4f6c5dd700 1 -- start start 2023-11-08T08:19:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f50003a00 0x7f4f64158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 0x7f4f6415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f6415c2b0 0x7f4f640c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6415c720 con 0x7f4f640c47a0 2023-11-08T08:19:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640c4c10 con 0x7f4f6415c2b0 2023-11-08T08:19:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640c3700 con 0x7f4f50003a00 2023-11-08T08:19:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 0x7f4f6415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 0x7f4f6415bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35862/0 (socket says 172.21.15.116:35862) 2023-11-08T08:19:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 -- 172.21.15.116:0/3805839694 learned_addr learned my addr 172.21.15.116:0/3805839694 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6a379700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f50003a00 0x7f4f64158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6ab7a700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f6415c2b0 0x7f4f640c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 -- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f50003a00 msgr2=0x7f4f64158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f50003a00 0x7f4f64158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 -- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f6415c2b0 msgr2=0x7f4f640c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f6415c2b0 0x7f4f640c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 -- 172.21.15.116:0/3805839694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6415a350 con 0x7f4f640c47a0 2023-11-08T08:19:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f69b78700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 0x7f4f6415bd70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f60011aa0 tx=0x7f4f60011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000f980 con 0x7f4f640c47a0 2023-11-08T08:19:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6000fae0 con 0x7f4f640c47a0 2023-11-08T08:19:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6415d590 con 0x7f4f640c47a0 2023-11-08T08:19:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000fe90 con 0x7f4f640c47a0 2023-11-08T08:19:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.130+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6415a4e0 con 0x7f4f640c47a0 2023-11-08T08:19:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.131+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f640743f0 con 0x7f4f640c47a0 2023-11-08T08:19:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.132+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f60014440 con 0x7f4f640c47a0 2023-11-08T08:19:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.132+0000 7f4f5affd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4f480682f0 0x7f4f4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.132+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4f60086430 con 0x7f4f640c47a0 2023-11-08T08:19:28.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.132+0000 7f4f6a379700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4f480682f0 0x7f4f4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.132+0000 7f4f6a379700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4f480682f0 0x7f4f4806a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f54014040 tx=0x7f4f5400a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.136+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f60052c30 con 0x7f4f640c47a0 2023-11-08T08:19:28.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.316+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4f6406e620 con 0x7f4f640c47a0 2023-11-08T08:19:28.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.316+0000 7f4f5affd700 1 -- 172.21.15.116:0/3805839694 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4f60056250 con 0x7f4f640c47a0 2023-11-08T08:19:28.318 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:28.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4f480682f0 msgr2=0x7f4f4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4f480682f0 0x7f4f4806a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f54014040 tx=0x7f4f5400a000 comp rx=0 tx=0).stop 2023-11-08T08:19:28.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 msgr2=0x7f4f6415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 0x7f4f6415bd70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f60011aa0 tx=0x7f4f60011e60 comp rx=0 tx=0).stop 2023-11-08T08:19:28.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 shutdown_connections 2023-11-08T08:19:28.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4f480682f0 0x7f4f4806a7a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f6415c2b0 0x7f4f640c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4f50003a00 0x7f4f64158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 --2- 172.21.15.116:0/3805839694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f640c47a0 0x7f4f6415bd70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.318+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 >> 172.21.15.116:0/3805839694 conn(0x7f4f6406c010 msgr2=0x7f4f640afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:28.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.319+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 shutdown_connections 2023-11-08T08:19:28.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.319+0000 7f4f6c5dd700 1 -- 172.21.15.116:0/3805839694 wait complete. 2023-11-08T08:19:28.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.455+0000 7fed93da6700 1 Processor -- start 2023-11-08T08:19:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.455+0000 7fed93da6700 1 -- start start 2023-11-08T08:19:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed93da6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 0x7fed8c06c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed93da6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fed8c06cdc0 0x7fed8c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed93da6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8c160a50 con 0x7fed8c15c810 2023-11-08T08:19:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed93da6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c15cbe0 con 0x7fed8c06cdc0 2023-11-08T08:19:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed93da6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c160d30 con 0x7fed8c0718e0 2023-11-08T08:19:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed91b42700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fed8c15c810 0x7fed8c160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56354/0 (socket says 172.21.15.116:56354) 2023-11-08T08:19:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed91b42700 1 -- 172.21.15.116:0/3057085962 learned_addr learned my addr 172.21.15.116:0/3057085962 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed91341700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 0x7fed8c06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed92343700 1 --2- 172.21.15.116:0/3057085962 >> v2:172.21.15.129:3300/0 conn(0x7fed8c06cdc0 0x7fed8c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165283807 0 0) 0x7fed8c160a50 con 0x7fed8c15c810 2023-11-08T08:19:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.456+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed780035b0 con 0x7fed8c15c810 2023-11-08T08:19:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245110662 0 0) 0x7fed780035b0 con 0x7fed8c15c810 2023-11-08T08:19:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 msgr2=0x7fed8c06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 0x7fed8c06c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 >> v2:172.21.15.129:3300/0 conn(0x7fed8c06cdc0 msgr2=0x7fed8c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 --2- 172.21.15.116:0/3057085962 >> v2:172.21.15.129:3300/0 conn(0x7fed8c06cdc0 0x7fed8c06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c161010 con 0x7fed8c15c810 2023-11-08T08:19:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fed7c001670 con 0x7fed8c15c810 2023-11-08T08:19:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 >> v1:172.21.15.116:6789/0 conn(0x7fed8c15c810 legacy=0x7fed8c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed78003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004430 0x7fed780048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 0x7fed78004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c15cbe0 con 0x7fed78003a00 2023-11-08T08:19:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c160d30 con 0x7fed78004430 2023-11-08T08:19:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7800ba10 con 0x7fed8c0718e0 2023-11-08T08:19:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed78003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91341700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004430 0x7fed780048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed92343700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 0x7fed78004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 -- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 msgr2=0x7fed78004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c0718e0 0x7fed78004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 -- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004430 msgr2=0x7fed780048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed78004430 0x7fed780048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 -- 172.21.15.116:0/3057085962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780035b0 con 0x7fed78003a00 2023-11-08T08:19:28.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed91b42700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed78003e10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c320 tx=0x7fed78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be9787d6a26613e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00ce60 con 0x7fed78003a00 2023-11-08T08:19:28.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c005960 con 0x7fed78003a00 2023-11-08T08:19:28.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.457+0000 7fed90b40700 1 -- 172.21.15.116:0/3057085962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c005cf0 con 0x7fed78003a00 2023-11-08T08:19:28.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed93da6700 1 -- 172.21.15.116:0/3057085962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 msgr2=0x7fed78003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed93da6700 1 --2- 172.21.15.116:0/3057085962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed78003e10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c320 tx=0x7fed78005880 comp rx=0 tx=0).stop 2023-11-08T08:19:28.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed92343700 1 -- 172.21.15.116:0/3057085962 reap_dead start 2023-11-08T08:19:28.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed93da6700 1 -- 172.21.15.116:0/3057085962 shutdown_connections 2023-11-08T08:19:28.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed93da6700 1 -- 172.21.15.116:0/3057085962 >> 172.21.15.116:0/3057085962 conn(0x7fed8c06c010 msgr2=0x7fed8c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:28.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed93da6700 1 -- 172.21.15.116:0/3057085962 shutdown_connections 2023-11-08T08:19:28.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.458+0000 7fed93da6700 1 -- 172.21.15.116:0/3057085962 wait complete. 2023-11-08T08:19:28.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.459+0000 7fed93da6700 1 Processor -- start 2023-11-08T08:19:28.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.459+0000 7fed93da6700 1 -- start start 2023-11-08T08:19:28.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed8c0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c15fca0 0x7fed8c15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c15f0a0 0x7fed8c1f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c145780 con 0x7fed78003a00 2023-11-08T08:19:28.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c0b52b0 con 0x7fed8c15fca0 2023-11-08T08:19:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c144e50 con 0x7fed8c15f0a0 2023-11-08T08:19:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed8c0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed8c0b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35878/0 (socket says 172.21.15.116:35878) 2023-11-08T08:19:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 -- 172.21.15.116:0/3534380399 learned_addr learned my addr 172.21.15.116:0/3534380399 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91341700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c15fca0 0x7fed8c15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed92343700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c15f0a0 0x7fed8c1f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 -- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c15f0a0 msgr2=0x7fed8c1f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c15f0a0 0x7fed8c1f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 -- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c15fca0 msgr2=0x7fed8c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c15fca0 0x7fed8c15eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 -- 172.21.15.116:0/3534380399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c0ce740 con 0x7fed78003a00 2023-11-08T08:19:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed91b42700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed8c0b1710 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c910 tx=0x7fed7c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c004870 con 0x7fed78003a00 2023-11-08T08:19:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c0049d0 con 0x7fed78003a00 2023-11-08T08:19:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c0ceb20 con 0x7fed78003a00 2023-11-08T08:19:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.460+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c004d80 con 0x7fed78003a00 2023-11-08T08:19:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.461+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c0ce8d0 con 0x7fed78003a00 2023-11-08T08:19:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.461+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c145a60 con 0x7fed78003a00 2023-11-08T08:19:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.462+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed7c024070 con 0x7fed78003a00 2023-11-08T08:19:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.462+0000 7fed827fc700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed700682f0 0x7fed7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.462+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fed7c086fc0 con 0x7fed78003a00 2023-11-08T08:19:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.462+0000 7fed91341700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed700682f0 0x7fed7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.463+0000 7fed91341700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed700682f0 0x7fed7006a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fed8c1f52d0 tx=0x7fed88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.466+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed7c0538e0 con 0x7fed78003a00 2023-11-08T08:19:28.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.647+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed8c0743f0 con 0x7fed78003a00 2023-11-08T08:19:28.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.647+0000 7fed827fc700 1 -- 172.21.15.116:0/3534380399 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed7c056f00 con 0x7fed78003a00 2023-11-08T08:19:28.649 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.649+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed700682f0 msgr2=0x7fed7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.649+0000 7fed93da6700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed700682f0 0x7fed7006a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fed8c1f52d0 tx=0x7fed88008040 comp rx=0 tx=0).stop 2023-11-08T08:19:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 msgr2=0x7fed8c0b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed8c0b1710 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c910 tx=0x7fed7c001a00 comp rx=0 tx=0).stop 2023-11-08T08:19:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 shutdown_connections 2023-11-08T08:19:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fed8c15f0a0 0x7fed8c1f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed78003a00 0x7fed8c0b1710 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fed700682f0 0x7fed7006a7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 --2- 172.21.15.116:0/3534380399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c15fca0 0x7fed8c15eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 >> 172.21.15.116:0/3534380399 conn(0x7fed8c06c010 msgr2=0x7fed8c0ad3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 shutdown_connections 2023-11-08T08:19:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.650+0000 7fed93da6700 1 -- 172.21.15.116:0/3534380399 wait complete. 2023-11-08T08:19:28.662 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:28.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.785+0000 7f80597bf700 1 Processor -- start 2023-11-08T08:19:28.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.785+0000 7f80597bf700 1 -- start start 2023-11-08T08:19:28.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80597bf700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f805415c8c0 0x7f8054160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80597bf700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f80540b5dc0 0x7f805406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80597bf700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540d2d30 con 0x7f805406b050 2023-11-08T08:19:28.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80597bf700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805406b4d0 con 0x7f80540b5dc0 2023-11-08T08:19:28.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80597bf700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80540d0b40 con 0x7f805415c8c0 2023-11-08T08:19:28.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80537fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f805406b050 0x7f80540d0fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56366/0 (socket says 172.21.15.116:56366) 2023-11-08T08:19:28.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80537fe700 1 -- 172.21.15.116:0/612224810 learned_addr learned my addr 172.21.15.116:0/612224810 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:28.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8052ffd700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f805415c8c0 0x7f8054160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80527fc700 1 --2- 172.21.15.116:0/612224810 >> v2:172.21.15.129:3300/0 conn(0x7f80540b5dc0 0x7f805406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391266303 0 0) 0x7f804c001490 con 0x7f805406b050 2023-11-08T08:19:28.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c0035b0 con 0x7f805406b050 2023-11-08T08:19:28.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176629143 0 0) 0x7f803c0035b0 con 0x7f805406b050 2023-11-08T08:19:28.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f805415c8c0 msgr2=0x7f8054160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f805415c8c0 0x7f8054160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 >> v2:172.21.15.129:3300/0 conn(0x7f80540b5dc0 msgr2=0x7f805406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 --2- 172.21.15.116:0/612224810 >> v2:172.21.15.129:3300/0 conn(0x7f80540b5dc0 0x7f805406ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80541611f0 con 0x7f805406b050 2023-11-08T08:19:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f804c001b80 con 0x7f805406b050 2023-11-08T08:19:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 >> v1:172.21.15.116:6789/0 conn(0x7f805406b050 legacy=0x7f80540d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c003a00 0x7f803c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f803c004430 0x7f803c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c004f60 0x7f803c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805406b4d0 con 0x7f803c003a00 2023-11-08T08:19:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80540d0b40 con 0x7f803c004430 2023-11-08T08:19:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c00bf50 con 0x7f803c004f60 2023-11-08T08:19:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8052ffd700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f805415c8c0 0x7f8054160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f8052ffd700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c003a00 0x7f803c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80527fc700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f803c004430 0x7f803c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.786+0000 7f80537fe700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c004f60 0x7f803c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8052ffd700 1 -- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c004f60 msgr2=0x7f803c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8052ffd700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c004f60 0x7f803c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8052ffd700 1 -- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f803c004430 msgr2=0x7f803c0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8052ffd700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f803c004430 0x7f803c0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8052ffd700 1 -- 172.21.15.116:0/612224810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c0035b0 con 0x7f803c003a00 2023-11-08T08:19:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f80527fc700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f803c004430 0x7f803c0048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8052ffd700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c003a00 0x7f803c003e10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f805408b520 tx=0x7f803c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c3f47a9ac32a2a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048004030 con 0x7f803c003a00 2023-11-08T08:19:28.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8048004190 con 0x7f803c003a00 2023-11-08T08:19:28.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.787+0000 7f8051ffb700 1 -- 172.21.15.116:0/612224810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048004540 con 0x7f803c003a00 2023-11-08T08:19:28.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80597bf700 1 -- 172.21.15.116:0/612224810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c003a00 msgr2=0x7f803c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80597bf700 1 --2- 172.21.15.116:0/612224810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c003a00 0x7f803c003e10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f805408b520 tx=0x7f803c005880 comp rx=0 tx=0).stop 2023-11-08T08:19:28.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80537fe700 1 -- 172.21.15.116:0/612224810 reap_dead start 2023-11-08T08:19:28.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80597bf700 1 -- 172.21.15.116:0/612224810 shutdown_connections 2023-11-08T08:19:28.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80597bf700 1 -- 172.21.15.116:0/612224810 >> 172.21.15.116:0/612224810 conn(0x7f805406c520 msgr2=0x7f805406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:28.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80597bf700 1 -- 172.21.15.116:0/612224810 shutdown_connections 2023-11-08T08:19:28.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.788+0000 7f80597bf700 1 -- 172.21.15.116:0/612224810 wait complete. 2023-11-08T08:19:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.789+0000 7f80597bf700 1 Processor -- start 2023-11-08T08:19:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.789+0000 7f80597bf700 1 -- start start 2023-11-08T08:19:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c003a00 0x7f80540cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 0x7f80541e8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80540ca570 0x7f80540ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80541e8c40 con 0x7f80541e8290 2023-11-08T08:19:28.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805415d000 con 0x7f80540ca570 2023-11-08T08:19:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80540cc1b0 con 0x7f803c003a00 2023-11-08T08:19:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 0x7f80541e8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 0x7f80541e8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35904/0 (socket says 172.21.15.116:35904) 2023-11-08T08:19:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 -- 172.21.15.116:0/3326510239 learned_addr learned my addr 172.21.15.116:0/3326510239 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f8052ffd700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c003a00 0x7f80540cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80537fe700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80540ca570 0x7f80540ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 -- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c003a00 msgr2=0x7f80540cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c003a00 0x7f80540cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 -- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80540ca570 msgr2=0x7f80540ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80540ca570 0x7f80540ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 -- 172.21.15.116:0/3326510239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540cd760 con 0x7f80541e8290 2023-11-08T08:19:28.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80527fc700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 0x7f80541e8700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8044011d00 tx=0x7f804400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804400fa80 con 0x7f80541e8290 2023-11-08T08:19:28.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804400fbe0 con 0x7f80541e8290 2023-11-08T08:19:28.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80540d2a50 con 0x7f80541e8290 2023-11-08T08:19:28.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044004020 con 0x7f80541e8290 2023-11-08T08:19:28.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.790+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80540cda40 con 0x7f80541e8290 2023-11-08T08:19:28.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.791+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8054067d70 con 0x7f80541e8290 2023-11-08T08:19:28.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.792+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8044024070 con 0x7f80541e8290 2023-11-08T08:19:28.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.792+0000 7f803b7fe700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80400682f0 0x7f804006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.792+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8044085f90 con 0x7f80541e8290 2023-11-08T08:19:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.792+0000 7f8052ffd700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80400682f0 0x7f804006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.793+0000 7f8052ffd700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80400682f0 0x7f804006a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8048016040 tx=0x7f804801f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.797+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80440527f0 con 0x7f80541e8290 2023-11-08T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.973+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80540743f0 con 0x7f80541e8290 2023-11-08T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.973+0000 7f803b7fe700 1 -- 172.21.15.116:0/3326510239 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80440220a0 con 0x7f80541e8290 2023-11-08T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80400682f0 msgr2=0x7f804006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80400682f0 0x7f804006a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8048016040 tx=0x7f804801f040 comp rx=0 tx=0).stop 2023-11-08T08:19:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 msgr2=0x7f80541e8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 0x7f80541e8700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8044011d00 tx=0x7f804400f520 comp rx=0 tx=0).stop 2023-11-08T08:19:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 shutdown_connections 2023-11-08T08:19:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80400682f0 0x7f804006a7a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80540ca570 0x7f80540ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f803c003a00 0x7f80540cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 --2- 172.21.15.116:0/3326510239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80541e8290 0x7f80541e8700 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 >> 172.21.15.116:0/3326510239 conn(0x7f805406c520 msgr2=0x7f80540b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 shutdown_connections 2023-11-08T08:19:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:28.975+0000 7f80597bf700 1 -- 172.21.15.116:0/3326510239 wait complete. 2023-11-08T08:19:28.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.111+0000 7fbbdfd16700 1 Processor -- start 2023-11-08T08:19:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.111+0000 7fbbdfd16700 1 -- start start 2023-11-08T08:19:29.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbdfd16700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbdfd16700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbd806b220 0x7fbbd80d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbdfd16700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd80d2d30 con 0x7fbbd80b5dc0 2023-11-08T08:19:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbdfd16700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd80d0b40 con 0x7fbbd806b220 2023-11-08T08:19:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbdfd16700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd81611f0 con 0x7fbbd815c8c0 2023-11-08T08:19:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbd806b220 0x7fbbd80d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbd806b220 0x7fbbd80d0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35914/0 (socket says 172.21.15.116:35914) 2023-11-08T08:19:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 -- 172.21.15.116:0/1726824833 learned_addr learned my addr 172.21.15.116:0/1726824833 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbddab2700 1 --2- 172.21.15.116:0/1726824833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 -- 172.21.15.116:0/1726824833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 msgr2=0x7fbbd8160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 --2- 172.21.15.116:0/1726824833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 -- 172.21.15.116:0/1726824833 >> v1:172.21.15.129:6789/0 conn(0x7fbbd80b5dc0 legacy=0x7fbbd806ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.112+0000 7fbbde2b3700 1 -- 172.21.15.116:0/1726824833 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd80d2a50 con 0x7fbbd806b220 2023-11-08T08:19:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbde2b3700 1 --2- 172.21.15.116:0/1726824833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd80d0fd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009e50 tx=0x7fbbd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61c0b6bddf60726 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdcab0700 1 -- 172.21.15.116:0/1726824833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd000e070 con 0x7fbbd806b220 2023-11-08T08:19:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdcab0700 1 -- 172.21.15.116:0/1726824833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd000b910 con 0x7fbbd806b220 2023-11-08T08:19:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdcab0700 1 -- 172.21.15.116:0/1726824833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd000bcc0 con 0x7fbbd806b220 2023-11-08T08:19:29.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 -- 172.21.15.116:0/1726824833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 msgr2=0x7fbbd80d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/1726824833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd80d0fd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009e50 tx=0x7fbbd000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:29.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 -- 172.21.15.116:0/1726824833 shutdown_connections 2023-11-08T08:19:29.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/1726824833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd80d0fd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/1726824833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 -- 172.21.15.116:0/1726824833 >> 172.21.15.116:0/1726824833 conn(0x7fbbd806c520 msgr2=0x7fbbd806c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:29.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 -- 172.21.15.116:0/1726824833 shutdown_connections 2023-11-08T08:19:29.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.113+0000 7fbbdfd16700 1 -- 172.21.15.116:0/1726824833 wait complete. 2023-11-08T08:19:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 Processor -- start 2023-11-08T08:19:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 -- start start 2023-11-08T08:19:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd814d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbd80b5dc0 0x7fbbd8156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80c7840 con 0x7fbbd806b220 2023-11-08T08:19:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80c7b20 con 0x7fbbd80b5dc0 2023-11-08T08:19:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbdfd16700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80c8190 con 0x7fbbd815c8c0 2023-11-08T08:19:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbddab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd814d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbddab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd814d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35930/0 (socket says 172.21.15.116:35930) 2023-11-08T08:19:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.115+0000 7fbbddab2700 1 -- 172.21.15.116:0/3870063608 learned_addr learned my addr 172.21.15.116:0/3870063608 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbddab2700 1 -- 172.21.15.116:0/3870063608 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 msgr2=0x7fbbd8157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbddab2700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbddab2700 1 -- 172.21.15.116:0/3870063608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbd80b5dc0 msgr2=0x7fbbd8156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbddab2700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbd80b5dc0 0x7fbbd8156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbddab2700 1 -- 172.21.15.116:0/3870063608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd0009b30 con 0x7fbbd806b220 2023-11-08T08:19:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbddab2700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd814d5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbbc800e870 tx=0x7fbbc800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc800ee70 con 0x7fbbd806b220 2023-11-08T08:19:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc800c850 con 0x7fbbd806b220 2023-11-08T08:19:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd80cda40 con 0x7fbbd806b220 2023-11-08T08:19:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc800cbc0 con 0x7fbbd806b220 2023-11-08T08:19:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.116+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd80cd760 con 0x7fbbd806b220 2023-11-08T08:19:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.117+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd80d2700 con 0x7fbbd806b220 2023-11-08T08:19:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.117+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbbc8019030 con 0x7fbbd806b220 2023-11-08T08:19:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.118+0000 7fbbce7fc700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbbbc0682f0 0x7fbbbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.118+0000 7fbbdd2b1700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbbbc0682f0 0x7fbbbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.118+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbbc8085b40 con 0x7fbbd806b220 2023-11-08T08:19:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.118+0000 7fbbdd2b1700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbbbc0682f0 0x7fbbbc06a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4000f60 tx=0x7fbbd4007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.122+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbbc80523a0 con 0x7fbbd806b220 2023-11-08T08:19:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.302+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbbd8067d70 con 0x7fbbd806b220 2023-11-08T08:19:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.302+0000 7fbbce7fc700 1 -- 172.21.15.116:0/3870063608 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbbc80170a0 con 0x7fbbd806b220 2023-11-08T08:19:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbbbc0682f0 msgr2=0x7fbbbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbbbc0682f0 0x7fbbbc06a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4000f60 tx=0x7fbbd4007a40 comp rx=0 tx=0).stop 2023-11-08T08:19:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 msgr2=0x7fbbd814d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd814d5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbbc800e870 tx=0x7fbbc800ec30 comp rx=0 tx=0).stop 2023-11-08T08:19:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 shutdown_connections 2023-11-08T08:19:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbd815c8c0 0x7fbbd8157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd806b220 0x7fbbd814d5d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbbbc0682f0 0x7fbbbc06a7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 --2- 172.21.15.116:0/3870063608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbd80b5dc0 0x7fbbd8156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 >> 172.21.15.116:0/3870063608 conn(0x7fbbd806c520 msgr2=0x7fbbd80b8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 shutdown_connections 2023-11-08T08:19:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.304+0000 7fbbdfd16700 1 -- 172.21.15.116:0/3870063608 wait complete. 2023-11-08T08:19:29.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.438+0000 7fd924069700 1 Processor -- start 2023-11-08T08:19:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.438+0000 7fd924069700 1 -- start start 2023-11-08T08:19:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd924069700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd91c06a980 0x7fd91c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd924069700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 0x7fd91c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd924069700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c15cc10 con 0x7fd91c0cf710 2023-11-08T08:19:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd924069700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c06c9b0 con 0x7fd91c06a980 2023-11-08T08:19:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd924069700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c06cc90 con 0x7fd91c071890 2023-11-08T08:19:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 0x7fd91c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 0x7fd91c071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:44812/0 (socket says 172.21.15.116:44812) 2023-11-08T08:19:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 -- 172.21.15.116:0/4061114503 learned_addr learned my addr 172.21.15.116:0/4061114503 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd921604700 1 --2- 172.21.15.116:0/4061114503 >> v2:172.21.15.129:3300/0 conn(0x7fd91c06a980 0x7fd91c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 -- 172.21.15.116:0/4061114503 >> v2:172.21.15.129:3300/0 conn(0x7fd91c06a980 msgr2=0x7fd91c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 --2- 172.21.15.116:0/4061114503 >> v2:172.21.15.129:3300/0 conn(0x7fd91c06a980 0x7fd91c0ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 -- 172.21.15.116:0/4061114503 >> v1:172.21.15.129:6789/0 conn(0x7fd91c0cf710 legacy=0x7fd91c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd922606700 1 -- 172.21.15.116:0/4061114503 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c06cf70 con 0x7fd91c071890 2023-11-08T08:19:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.439+0000 7fd920e03700 1 -- 172.21.15.116:0/4061114503 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770453174 0 0) 0x7fd91c15cc10 con 0x7fd91c0cf710 2023-11-08T08:19:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd922606700 1 --2- 172.21.15.116:0/4061114503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 0x7fd91c071220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd914009e50 tx=0x7fd91400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17916474366f0101 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd920e03700 1 -- 172.21.15.116:0/4061114503 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400e070 con 0x7fd91c071890 2023-11-08T08:19:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd920e03700 1 -- 172.21.15.116:0/4061114503 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91400b910 con 0x7fd91c071890 2023-11-08T08:19:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd920e03700 1 -- 172.21.15.116:0/4061114503 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400bcc0 con 0x7fd91c071890 2023-11-08T08:19:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 -- 172.21.15.116:0/4061114503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 msgr2=0x7fd91c071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 --2- 172.21.15.116:0/4061114503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 0x7fd91c071220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd914009e50 tx=0x7fd91400b330 comp rx=0 tx=0).stop 2023-11-08T08:19:29.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 -- 172.21.15.116:0/4061114503 shutdown_connections 2023-11-08T08:19:29.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 --2- 172.21.15.116:0/4061114503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c071890 0x7fd91c071220 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 --2- 172.21.15.116:0/4061114503 >> v2:172.21.15.129:3300/0 conn(0x7fd91c06a980 0x7fd91c0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 -- 172.21.15.116:0/4061114503 >> 172.21.15.116:0/4061114503 conn(0x7fd91c06c010 msgr2=0x7fd91c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:29.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 -- 172.21.15.116:0/4061114503 shutdown_connections 2023-11-08T08:19:29.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.440+0000 7fd924069700 1 -- 172.21.15.116:0/4061114503 wait complete. 2023-11-08T08:19:29.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.442+0000 7fd924069700 1 Processor -- start 2023-11-08T08:19:29.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.442+0000 7fd924069700 1 -- start start 2023-11-08T08:19:29.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd924069700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c06a980 0x7fd91c1f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd924069700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd91c071890 0x7fd91c15fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd924069700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 0x7fd91c15edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd924069700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c15d170 con 0x7fd91c0cf710 2023-11-08T08:19:29.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd924069700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c1f68d0 con 0x7fd91c071890 2023-11-08T08:19:29.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd924069700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c1f2390 con 0x7fd91c06a980 2023-11-08T08:19:29.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 0x7fd91c15edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 0x7fd91c15edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35932/0 (socket says 172.21.15.116:35932) 2023-11-08T08:19:29.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 -- 172.21.15.116:0/2670226851 learned_addr learned my addr 172.21.15.116:0/2670226851 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:29.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd921604700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd91c071890 0x7fd91c15fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd921e05700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c06a980 0x7fd91c1f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 -- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c06a980 msgr2=0x7fd91c1f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c06a980 0x7fd91c1f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 -- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd91c071890 msgr2=0x7fd91c15fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd91c071890 0x7fd91c15fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 -- 172.21.15.116:0/2670226851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914009b30 con 0x7fd91c0cf710 2023-11-08T08:19:29.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.443+0000 7fd922606700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 0x7fd91c15edf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd91c06cdd0 tx=0x7fd914004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.444+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914003ea0 con 0x7fd91c0cf710 2023-11-08T08:19:29.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.444+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9140048e0 con 0x7fd91c0cf710 2023-11-08T08:19:29.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.444+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c06d530 con 0x7fd91c0cf710 2023-11-08T08:19:29.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.444+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91401c680 con 0x7fd91c0cf710 2023-11-08T08:19:29.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.444+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c06d250 con 0x7fd91c0cf710 2023-11-08T08:19:29.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.445+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c160480 con 0x7fd91c0cf710 2023-11-08T08:19:29.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.445+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd91402c070 con 0x7fd91c0cf710 2023-11-08T08:19:29.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.445+0000 7fd9127fc700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd9000682f0 0x7fd90006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.445+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd914087760 con 0x7fd91c0cf710 2023-11-08T08:19:29.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.445+0000 7fd921e05700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd9000682f0 0x7fd90006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.446+0000 7fd921e05700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd9000682f0 0x7fd90006a7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd918000f60 tx=0x7fd918009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.449+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd914053fc0 con 0x7fd91c0cf710 2023-11-08T08:19:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.646+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd91c1f2560 con 0x7fd91c0cf710 2023-11-08T08:19:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.646+0000 7fd9127fc700 1 -- 172.21.15.116:0/2670226851 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd9140575e0 con 0x7fd91c0cf710 2023-11-08T08:19:29.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd9000682f0 msgr2=0x7fd90006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd9000682f0 0x7fd90006a7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd918000f60 tx=0x7fd918009040 comp rx=0 tx=0).stop 2023-11-08T08:19:29.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 msgr2=0x7fd91c15edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 0x7fd91c15edf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd91c06cdd0 tx=0x7fd914004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 shutdown_connections 2023-11-08T08:19:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91c0cf710 0x7fd91c15edf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd91c06a980 0x7fd91c1f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd9000682f0 0x7fd90006a7a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 --2- 172.21.15.116:0/2670226851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd91c071890 0x7fd91c15fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 >> 172.21.15.116:0/2670226851 conn(0x7fd91c06c010 msgr2=0x7fd91c0d0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 shutdown_connections 2023-11-08T08:19:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.648+0000 7fd924069700 1 -- 172.21.15.116:0/2670226851 wait complete. 2023-11-08T08:19:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:29.662 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi129.front.sepia.ceph.com... 2023-11-08T08:19:29.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:29.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-11-08T08:19:29.709 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:29.710 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:19:29.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:29.711 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-08T08:19:29.731 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-08T08:19:29.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:19:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.851+0000 7feb142eb700 1 Processor -- start 2023-11-08T08:19:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.851+0000 7feb142eb700 1 -- start start 2023-11-08T08:19:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.851+0000 7feb142eb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb0c0b0880 0x7feb0c06b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb142eb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c0ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb142eb700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0adb90 con 0x7feb0c072180 2023-11-08T08:19:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb142eb700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c072620 con 0x7feb0c0b0880 2023-11-08T08:19:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb142eb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0b43f0 con 0x7feb0c0ad1f0 2023-11-08T08:19:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb12087700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb0c0b0880 0x7feb0c06b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb11886700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feb0c072180 0x7feb0c0aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56382/0 (socket says 172.21.15.116:56382) 2023-11-08T08:19:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb11886700 1 -- 172.21.15.116:0/1970649417 learned_addr learned my addr 172.21.15.116:0/1970649417 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb12087700 1 -- 172.21.15.116:0/1970649417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 msgr2=0x7feb0c0ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb12087700 1 --2- 172.21.15.116:0/1970649417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c0ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb12087700 1 -- 172.21.15.116:0/1970649417 >> v1:172.21.15.116:6789/0 conn(0x7feb0c072180 legacy=0x7feb0c0aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb11085700 1 -- 172.21.15.116:0/1970649417 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663162724 0 0) 0x7feb0c0adb90 con 0x7feb0c072180 2023-11-08T08:19:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb12087700 1 -- 172.21.15.116:0/1970649417 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c0b46d0 con 0x7feb0c0b0880 2023-11-08T08:19:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb12087700 1 --2- 172.21.15.116:0/1970649417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0b0880 0x7feb0c06b7b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7feafc009840 tx=0x7feafc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90f7c3eeb11f8d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb11085700 1 -- 172.21.15.116:0/1970649417 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc011830 con 0x7feb0c0b0880 2023-11-08T08:19:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb11085700 1 -- 172.21.15.116:0/1970649417 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc011990 con 0x7feb0c0b0880 2023-11-08T08:19:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.852+0000 7feb11085700 1 -- 172.21.15.116:0/1970649417 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc011d20 con 0x7feb0c0b0880 2023-11-08T08:19:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 -- 172.21.15.116:0/1970649417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0b0880 msgr2=0x7feb0c06b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 --2- 172.21.15.116:0/1970649417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0b0880 0x7feb0c06b7b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7feafc009840 tx=0x7feafc009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 -- 172.21.15.116:0/1970649417 shutdown_connections 2023-11-08T08:19:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 --2- 172.21.15.116:0/1970649417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0b0880 0x7feb0c06b7b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 --2- 172.21.15.116:0/1970649417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c0ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 -- 172.21.15.116:0/1970649417 >> 172.21.15.116:0/1970649417 conn(0x7feb0c0b2fc0 msgr2=0x7feb0c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 -- 172.21.15.116:0/1970649417 shutdown_connections 2023-11-08T08:19:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.853+0000 7feb142eb700 1 -- 172.21.15.116:0/1970649417 wait complete. 2023-11-08T08:19:29.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.854+0000 7feb142eb700 1 Processor -- start 2023-11-08T08:19:29.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.854+0000 7feb142eb700 1 -- start start 2023-11-08T08:19:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb142eb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 0x7feb0c15fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb142eb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb142eb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb0c0b0880 0x7feb0c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb142eb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c15b470 con 0x7feb0c072180 2023-11-08T08:19:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb142eb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c157cd0 con 0x7feb0c0b0880 2023-11-08T08:19:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb142eb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c157fb0 con 0x7feb0c0ad1f0 2023-11-08T08:19:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 0x7feb0c15fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 0x7feb0c15fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35954/0 (socket says 172.21.15.116:35954) 2023-11-08T08:19:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 -- 172.21.15.116:0/331847959 learned_addr learned my addr 172.21.15.116:0/331847959 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:29.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb11886700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12888700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb0c0b0880 0x7feb0c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 -- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 msgr2=0x7feb0c15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c15dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 -- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb0c0b0880 msgr2=0x7feb0c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:29.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb0c0b0880 0x7feb0c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:29.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 -- 172.21.15.116:0/331847959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc0094a0 con 0x7feb0c072180 2023-11-08T08:19:29.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb11886700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:29.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12888700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb0c0b0880 0x7feb0c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:29.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb12087700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 0x7feb0c15fa60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7feafc0090b0 tx=0x7feafc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.855+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc0045f0 con 0x7feb0c072180 2023-11-08T08:19:29.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.856+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc01f070 con 0x7feb0c072180 2023-11-08T08:19:29.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.856+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c0b4fe0 con 0x7feb0c072180 2023-11-08T08:19:29.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.856+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc003b00 con 0x7feb0c072180 2023-11-08T08:19:29.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.856+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c0b4d00 con 0x7feb0c072180 2023-11-08T08:19:29.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.856+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c0b49b0 con 0x7feb0c072180 2023-11-08T08:19:29.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.857+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feafc00f050 con 0x7feb0c072180 2023-11-08T08:19:29.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.857+0000 7feb02ffd700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaf00682f0 0x7feaf006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:29.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.857+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feafc021070 con 0x7feb0c072180 2023-11-08T08:19:29.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.857+0000 7feb11886700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaf00682f0 0x7feaf006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:29.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.858+0000 7feb11886700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaf00682f0 0x7feaf006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feb08007fd0 tx=0x7feb08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:29.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:29.861+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feafc0540b0 con 0x7feb0c072180 2023-11-08T08:19:30.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.082+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb0c0c6290 con 0x7feb0c072180 2023-11-08T08:19:30.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.083+0000 7feb02ffd700 1 -- 172.21.15.116:0/331847959 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4654 (secure 0 0 0) 0x7feafc02b090 con 0x7feb0c072180 2023-11-08T08:19:30.084 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:30.085 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:18:42.885473+0000","modified":"2023-11-08T08:18:45.898738+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-08T08:19:30.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaf00682f0 msgr2=0x7feaf006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaf00682f0 0x7feaf006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feb08007fd0 tx=0x7feb08009040 comp rx=0 tx=0).stop 2023-11-08T08:19:30.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 msgr2=0x7feb0c15fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 0x7feb0c15fa60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7feafc0090b0 tx=0x7feafc004750 comp rx=0 tx=0).stop 2023-11-08T08:19:30.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 shutdown_connections 2023-11-08T08:19:30.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaf00682f0 0x7feaf006a7a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb0c0b0880 0x7feb0c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c072180 0x7feb0c15fa60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 --2- 172.21.15.116:0/331847959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0c0ad1f0 0x7feb0c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 >> 172.21.15.116:0/331847959 conn(0x7feb0c0b2fc0 msgr2=0x7feb0c06d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 shutdown_connections 2023-11-08T08:19:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.085+0000 7feb142eb700 1 -- 172.21.15.116:0/331847959 wait complete. 2023-11-08T08:19:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-08T08:19:30.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:30.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.217+0000 7f574468c700 1 Processor -- start 2023-11-08T08:19:30.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.217+0000 7f574468c700 1 -- start start 2023-11-08T08:19:30.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.218+0000 7f574468c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f573c06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.218+0000 7f574468c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f573c06e3a0 0x7f573c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.218+0000 7f574468c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0b1810 con 0x7f573c06a270 2023-11-08T08:19:30.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.218+0000 7f574468c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0b2490 con 0x7f573c06e3a0 2023-11-08T08:19:30.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.218+0000 7f574468c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0b1af0 con 0x7f573c15cc80 2023-11-08T08:19:30.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742428700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f573c06a270 0x7f573c06a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56394/0 (socket says 172.21.15.116:56394) 2023-11-08T08:19:30.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742428700 1 -- 172.21.15.116:0/2346020712 learned_addr learned my addr 172.21.15.116:0/2346020712 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:30.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392183967 0 0) 0x7f573c0b1810 con 0x7f573c06a270 2023-11-08T08:19:30.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 --2- 172.21.15.116:0/2346020712 >> v2:172.21.15.129:3300/0 conn(0x7f573c06e3a0 0x7f573c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57280035b0 con 0x7f573c06a270 2023-11-08T08:19:30.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741c27700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f573c06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1933901511 0 0) 0x7f57280035b0 con 0x7f573c06a270 2023-11-08T08:19:30.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 msgr2=0x7f573c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f573c06de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 >> v2:172.21.15.129:3300/0 conn(0x7f573c06e3a0 msgr2=0x7f573c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 --2- 172.21.15.116:0/2346020712 >> v2:172.21.15.129:3300/0 conn(0x7f573c06e3a0 0x7f573c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0b1dd0 con 0x7f573c06a270 2023-11-08T08:19:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f572c001670 con 0x7f573c06a270 2023-11-08T08:19:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741c27700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f573c06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 --2- 172.21.15.116:0/2346020712 >> v2:172.21.15.129:3300/0 conn(0x7f573c06e3a0 0x7f573c070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:30.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 >> v1:172.21.15.116:6789/0 conn(0x7f573c06a270 legacy=0x7f573c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f5728003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5728003ff0 0x7f5728004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f5728005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0b2490 con 0x7f5728004d90 2023-11-08T08:19:30.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0b1af0 con 0x7f5728003ff0 2023-11-08T08:19:30.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572800bb70 con 0x7f573c15cc80 2023-11-08T08:19:30.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f5728005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5741c27700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5728003ff0 0x7f5728004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742428700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f5728003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 -- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 msgr2=0x7f5728003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c15cc80 0x7f5728003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 -- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5728003ff0 msgr2=0x7f5728004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.219+0000 7f5742c29700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5728003ff0 0x7f5728004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f5742c29700 1 -- 172.21.15.116:0/2346020712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57280035b0 con 0x7f5728004d90 2023-11-08T08:19:30.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f5742c29700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f5728005200 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f57340073b0 tx=0x7f5728005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237da7a40f1c091c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734014400 con 0x7f5728004d90 2023-11-08T08:19:30.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734014560 con 0x7f5728004d90 2023-11-08T08:19:30.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f5741426700 1 -- 172.21.15.116:0/2346020712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340148f0 con 0x7f5728004d90 2023-11-08T08:19:30.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f574468c700 1 -- 172.21.15.116:0/2346020712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 msgr2=0x7f5728005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f574468c700 1 --2- 172.21.15.116:0/2346020712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f5728005200 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f57340073b0 tx=0x7f5728005880 comp rx=0 tx=0).stop 2023-11-08T08:19:30.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f5742c29700 1 -- 172.21.15.116:0/2346020712 reap_dead start 2023-11-08T08:19:30.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f574468c700 1 -- 172.21.15.116:0/2346020712 shutdown_connections 2023-11-08T08:19:30.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f574468c700 1 -- 172.21.15.116:0/2346020712 >> 172.21.15.116:0/2346020712 conn(0x7f573c06c010 msgr2=0x7f573c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:30.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f574468c700 1 -- 172.21.15.116:0/2346020712 shutdown_connections 2023-11-08T08:19:30.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.220+0000 7f574468c700 1 -- 172.21.15.116:0/2346020712 wait complete. 2023-11-08T08:19:30.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 Processor -- start 2023-11-08T08:19:30.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 -- start start 2023-11-08T08:19:30.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f573c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c0c6dc0 0x7f573c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f573c15a830 0x7f573c15acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c75f0 con 0x7f5728004d90 2023-11-08T08:19:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c78d0 con 0x7f573c15a830 2023-11-08T08:19:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f574468c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c7bb0 con 0x7f573c0c6dc0 2023-11-08T08:19:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f5742428700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f573c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f5742428700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f573c15be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35974/0 (socket says 172.21.15.116:35974) 2023-11-08T08:19:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f5742428700 1 -- 172.21.15.116:0/834311882 learned_addr learned my addr 172.21.15.116:0/834311882 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f5742c29700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f573c15a830 0x7f573c15acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.222+0000 7f5741c27700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c0c6dc0 0x7f573c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5742428700 1 -- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c0c6dc0 msgr2=0x7f573c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5742428700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c0c6dc0 0x7f573c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5742428700 1 -- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f573c15a830 msgr2=0x7f573c15acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5742428700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f573c15a830 0x7f573c15acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5742428700 1 -- 172.21.15.116:0/834311882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c06e7e0 con 0x7f5728004d90 2023-11-08T08:19:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5742428700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f573c15be50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f572c007310 tx=0x7f572c007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00f5e0 con 0x7f5728004d90 2023-11-08T08:19:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c00f780 con 0x7f5728004d90 2023-11-08T08:19:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c0b20b0 con 0x7f5728004d90 2023-11-08T08:19:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00fb30 con 0x7f5728004d90 2023-11-08T08:19:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.223+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c15daa0 con 0x7f5728004d90 2023-11-08T08:19:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.224+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c0743f0 con 0x7f5728004d90 2023-11-08T08:19:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.224+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f572c00fc90 con 0x7f5728004d90 2023-11-08T08:19:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.225+0000 7f5732ffd700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57200682f0 0x7f572006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.225+0000 7f5741c27700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57200682f0 0x7f572006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.225+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f572c085e20 con 0x7f5728004d90 2023-11-08T08:19:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.225+0000 7f5741c27700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57200682f0 0x7f572006a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f573c160120 tx=0x7f5738008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.229+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f572c052600 con 0x7f5728004d90 2023-11-08T08:19:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.413+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f573c0cd5f0 con 0x7f5728004d90 2023-11-08T08:19:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.414+0000 7f5732ffd700 1 -- 172.21.15.116:0/834311882 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f572c055c20 con 0x7f5728004d90 2023-11-08T08:19:30.415 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:30.416 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57200682f0 msgr2=0x7f572006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57200682f0 0x7f572006a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f573c160120 tx=0x7f5738008040 comp rx=0 tx=0).stop 2023-11-08T08:19:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 msgr2=0x7f573c15be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f573c15be50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f572c007310 tx=0x7f572c007620 comp rx=0 tx=0).stop 2023-11-08T08:19:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 shutdown_connections 2023-11-08T08:19:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57200682f0 0x7f572006a7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f573c15a830 0x7f573c15acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5728004d90 0x7f573c15be50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 --2- 172.21.15.116:0/834311882 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f573c0c6dc0 0x7f573c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 >> 172.21.15.116:0/834311882 conn(0x7f573c06c010 msgr2=0x7f573c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 shutdown_connections 2023-11-08T08:19:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.416+0000 7f574468c700 1 -- 172.21.15.116:0/834311882 wait complete. 2023-11-08T08:19:30.429 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:18:40.568047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:18:41.836252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:30.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.554+0000 7fc320320700 1 Processor -- start 2023-11-08T08:19:30.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.554+0000 7fc320320700 1 -- start start 2023-11-08T08:19:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc320320700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc318159710 0x7fc318070370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc320320700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3180c9a50 0x7fc3180c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc320320700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3180ca3f0 con 0x7fc3180cee40 2023-11-08T08:19:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc320320700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180cf2c0 con 0x7fc3180c9a50 2023-11-08T08:19:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc320320700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31806a310 con 0x7fc318159710 2023-11-08T08:19:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3180c9a50 0x7fc3180c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3180c9a50 0x7fc3180c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35988/0 (socket says 172.21.15.116:35988) 2023-11-08T08:19:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 -- 172.21.15.116:0/3353795793 learned_addr learned my addr 172.21.15.116:0/3353795793 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e0bc700 1 --2- 172.21.15.116:0/3353795793 >> v2:172.21.15.129:3300/0 conn(0x7fc318159710 0x7fc318070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 -- 172.21.15.116:0/3353795793 >> v2:172.21.15.129:3300/0 conn(0x7fc318159710 msgr2=0x7fc318070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 --2- 172.21.15.116:0/3353795793 >> v2:172.21.15.129:3300/0 conn(0x7fc318159710 0x7fc318070370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 -- 172.21.15.116:0/3353795793 >> v1:172.21.15.129:6789/0 conn(0x7fc3180cee40 legacy=0x7fc3180cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.555+0000 7fc31e8bd700 1 -- 172.21.15.116:0/3353795793 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180ce7e0 con 0x7fc3180c9a50 2023-11-08T08:19:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc31e8bd700 1 --2- 172.21.15.116:0/3353795793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3180c9a50 0x7fc3180c9eb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d150 tx=0x7fc31000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f5d621bda71628 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc31d0ba700 1 -- 172.21.15.116:0/3353795793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc310010070 con 0x7fc3180c9a50 2023-11-08T08:19:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc31d0ba700 1 -- 172.21.15.116:0/3353795793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31000dcd0 con 0x7fc3180c9a50 2023-11-08T08:19:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc31d0ba700 1 -- 172.21.15.116:0/3353795793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000b310 con 0x7fc3180c9a50 2023-11-08T08:19:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 -- 172.21.15.116:0/3353795793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3180c9a50 msgr2=0x7fc3180c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 --2- 172.21.15.116:0/3353795793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3180c9a50 0x7fc3180c9eb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d150 tx=0x7fc31000d460 comp rx=0 tx=0).stop 2023-11-08T08:19:30.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 -- 172.21.15.116:0/3353795793 shutdown_connections 2023-11-08T08:19:30.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 --2- 172.21.15.116:0/3353795793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3180c9a50 0x7fc3180c9eb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 --2- 172.21.15.116:0/3353795793 >> v2:172.21.15.129:3300/0 conn(0x7fc318159710 0x7fc318070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 -- 172.21.15.116:0/3353795793 >> 172.21.15.116:0/3353795793 conn(0x7fc3180cf400 msgr2=0x7fc3180cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:30.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 -- 172.21.15.116:0/3353795793 shutdown_connections 2023-11-08T08:19:30.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.556+0000 7fc320320700 1 -- 172.21.15.116:0/3353795793 wait complete. 2023-11-08T08:19:30.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.557+0000 7fc320320700 1 Processor -- start 2023-11-08T08:19:30.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 -- start start 2023-11-08T08:19:30.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3180c9a50 0x7fc318156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3180cee40 0x7fc3180b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 0x7fc3181561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc318156720 con 0x7fc318159710 2023-11-08T08:19:30.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3181e2dc0 con 0x7fc3180cee40 2023-11-08T08:19:30.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc320320700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3181570e0 con 0x7fc3180c9a50 2023-11-08T08:19:30.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc31e8bd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 0x7fc3181561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc31e8bd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 0x7fc3181561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36004/0 (socket says 172.21.15.116:36004) 2023-11-08T08:19:30.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc31e8bd700 1 -- 172.21.15.116:0/504446754 learned_addr learned my addr 172.21.15.116:0/504446754 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:30.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc31e0bc700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3180c9a50 0x7fc318156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.558+0000 7fc31d8bb700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3180cee40 0x7fc3180b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc31e8bd700 1 -- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3180c9a50 msgr2=0x7fc318156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc31e8bd700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3180c9a50 0x7fc318156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc31e8bd700 1 -- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3180cee40 msgr2=0x7fc3180b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc31e8bd700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3180cee40 0x7fc3180b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc31e8bd700 1 -- 172.21.15.116:0/504446754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31000bd40 con 0x7fc318159710 2023-11-08T08:19:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc31e8bd700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 0x7fc3181561e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d7b0 tx=0x7fc310004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000bbc0 con 0x7fc318159710 2023-11-08T08:19:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31000b4c0 con 0x7fc318159710 2023-11-08T08:19:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31806a890 con 0x7fc318159710 2023-11-08T08:19:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000aea0 con 0x7fc318159710 2023-11-08T08:19:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.559+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31806a5b0 con 0x7fc318159710 2023-11-08T08:19:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.560+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3180ceac0 con 0x7fc318159710 2023-11-08T08:19:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.560+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc31002d070 con 0x7fc318159710 2023-11-08T08:19:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.560+0000 7fc30effd700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc2fc0682f0 0x7fc2fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.561+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc31000de40 con 0x7fc318159710 2023-11-08T08:19:30.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.561+0000 7fc31e0bc700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc2fc0682f0 0x7fc2fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.561+0000 7fc31e0bc700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc2fc0682f0 0x7fc2fc06a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc3180c6030 tx=0x7fc308009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.565+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc310055b40 con 0x7fc318159710 2023-11-08T08:19:30.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.744+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc3180b4c90 con 0x7fc318159710 2023-11-08T08:19:30.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.744+0000 7fc30effd700 1 -- 172.21.15.116:0/504446754 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc310059160 con 0x7fc318159710 2023-11-08T08:19:30.746 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:30.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc2fc0682f0 msgr2=0x7fc2fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc2fc0682f0 0x7fc2fc06a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc3180c6030 tx=0x7fc308009210 comp rx=0 tx=0).stop 2023-11-08T08:19:30.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 msgr2=0x7fc3181561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 0x7fc3181561e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d7b0 tx=0x7fc310004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 shutdown_connections 2023-11-08T08:19:30.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc318159710 0x7fc3181561e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc3180c9a50 0x7fc318156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc2fc0682f0 0x7fc2fc06a7a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 --2- 172.21.15.116:0/504446754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3180cee40 0x7fc3180b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 >> 172.21.15.116:0/504446754 conn(0x7fc3180cf400 msgr2=0x7fc318071520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:30.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 shutdown_connections 2023-11-08T08:19:30.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.746+0000 7fc320320700 1 -- 172.21.15.116:0/504446754 wait complete. 2023-11-08T08:19:30.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:30.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.882+0000 7f81cd07d700 1 Processor -- start 2023-11-08T08:19:30.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.882+0000 7f81cd07d700 1 -- start start 2023-11-08T08:19:30.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81cd07d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f81c80cf710 0x7f81c80ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81cd07d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81c806a980 0x7f81c8071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81cd07d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c815cc10 con 0x7f81c8071890 2023-11-08T08:19:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81cd07d700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c806c9b0 con 0x7f81c806a980 2023-11-08T08:19:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81cd07d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c806cc90 con 0x7f81c80cf710 2023-11-08T08:19:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81c806a980 0x7f81c8071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81c806a980 0x7f81c8071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36008/0 (socket says 172.21.15.116:36008) 2023-11-08T08:19:30.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 -- 172.21.15.116:0/771100715 learned_addr learned my addr 172.21.15.116:0/771100715 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:30.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c6d9d700 1 --2- 172.21.15.116:0/771100715 >> v2:172.21.15.129:3300/0 conn(0x7f81c80cf710 0x7f81c80ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 -- 172.21.15.116:0/771100715 >> v2:172.21.15.129:3300/0 conn(0x7f81c80cf710 msgr2=0x7f81c80ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 --2- 172.21.15.116:0/771100715 >> v2:172.21.15.129:3300/0 conn(0x7f81c80cf710 0x7f81c80ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 -- 172.21.15.116:0/771100715 >> v1:172.21.15.129:6789/0 conn(0x7f81c8071890 legacy=0x7f81c806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:30.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.883+0000 7f81c759e700 1 -- 172.21.15.116:0/771100715 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c806cf70 con 0x7f81c806a980 2023-11-08T08:19:30.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81c759e700 1 --2- 172.21.15.116:0/771100715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c806a980 0x7f81c8071220 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009e50 tx=0x7f81c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ddce885cce4889 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81c5d9b700 1 -- 172.21.15.116:0/771100715 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c000e070 con 0x7f81c806a980 2023-11-08T08:19:30.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81c5d9b700 1 -- 172.21.15.116:0/771100715 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c0003a20 con 0x7f81c806a980 2023-11-08T08:19:30.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81c5d9b700 1 -- 172.21.15.116:0/771100715 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0003dd0 con 0x7f81c806a980 2023-11-08T08:19:30.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 -- 172.21.15.116:0/771100715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c806a980 msgr2=0x7f81c8071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 --2- 172.21.15.116:0/771100715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c806a980 0x7f81c8071220 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009e50 tx=0x7f81c000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:30.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 -- 172.21.15.116:0/771100715 shutdown_connections 2023-11-08T08:19:30.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 --2- 172.21.15.116:0/771100715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c806a980 0x7f81c8071220 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 --2- 172.21.15.116:0/771100715 >> v2:172.21.15.129:3300/0 conn(0x7f81c80cf710 0x7f81c80ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 -- 172.21.15.116:0/771100715 >> 172.21.15.116:0/771100715 conn(0x7f81c806c010 msgr2=0x7f81c80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:30.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 -- 172.21.15.116:0/771100715 shutdown_connections 2023-11-08T08:19:30.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.884+0000 7f81cd07d700 1 -- 172.21.15.116:0/771100715 wait complete. 2023-11-08T08:19:30.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 Processor -- start 2023-11-08T08:19:30.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 -- start start 2023-11-08T08:19:30.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c806a980 0x7f81c80b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81c8071890 0x7f81c8151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 0x7f81c80ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8158060 con 0x7f81c80cf710 2023-11-08T08:19:30.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.886+0000 7f81cd07d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8158340 con 0x7f81c806a980 2023-11-08T08:19:30.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81cd07d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8158620 con 0x7f81c8071890 2023-11-08T08:19:30.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 0x7f81c80ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 0x7f81c80ac8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36010/0 (socket says 172.21.15.116:36010) 2023-11-08T08:19:30.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 -- 172.21.15.116:0/1339860112 learned_addr learned my addr 172.21.15.116:0/1339860112 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:30.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c6d9d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c806a980 0x7f81c80b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c659c700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81c8071890 0x7f81c8151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 -- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81c8071890 msgr2=0x7f81c8151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81c8071890 0x7f81c8151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 -- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c806a980 msgr2=0x7f81c80b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:30.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c806a980 0x7f81c80b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 -- 172.21.15.116:0/1339860112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c0009b30 con 0x7f81c80cf710 2023-11-08T08:19:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81c759e700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 0x7f81c80ac8f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f81c0004000 tx=0x7f81c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0004440 con 0x7f81c80cf710 2023-11-08T08:19:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c00045c0 con 0x7f81c80cf710 2023-11-08T08:19:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c806d530 con 0x7f81c80cf710 2023-11-08T08:19:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0004970 con 0x7f81c80cf710 2023-11-08T08:19:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.887+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c806d250 con 0x7f81c80cf710 2023-11-08T08:19:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.888+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c8160480 con 0x7f81c80cf710 2023-11-08T08:19:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.889+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81c0022030 con 0x7f81c80cf710 2023-11-08T08:19:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.889+0000 7f81af7fe700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b40682f0 0x7f81b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.889+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f81c0086e00 con 0x7f81c80cf710 2023-11-08T08:19:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.889+0000 7f81c6d9d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b40682f0 0x7f81b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.889+0000 7f81c6d9d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b40682f0 0x7f81b406a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f81bc006b20 tx=0x7f81bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:30.893+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81c0053660 con 0x7f81c80cf710 2023-11-08T08:19:31.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.077+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81c80743f0 con 0x7f81c80cf710 2023-11-08T08:19:31.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.078+0000 7f81af7fe700 1 -- 172.21.15.116:0/1339860112 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81c00170a0 con 0x7f81c80cf710 2023-11-08T08:19:31.079 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.079+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b40682f0 msgr2=0x7f81b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.079+0000 7f81cd07d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b40682f0 0x7f81b406a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f81bc006b20 tx=0x7f81bc008040 comp rx=0 tx=0).stop 2023-11-08T08:19:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.079+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 msgr2=0x7f81c80ac8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.079+0000 7f81cd07d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 0x7f81c80ac8f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f81c0004000 tx=0x7f81c0004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 shutdown_connections 2023-11-08T08:19:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c80cf710 0x7f81c80ac8f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b40682f0 0x7f81b406a7a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c806a980 0x7f81c80b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 --2- 172.21.15.116:0/1339860112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81c8071890 0x7f81c8151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 >> 172.21.15.116:0/1339860112 conn(0x7f81c806c010 msgr2=0x7f81c806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:31.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 shutdown_connections 2023-11-08T08:19:31.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.080+0000 7f81cd07d700 1 -- 172.21.15.116:0/1339860112 wait complete. 2023-11-08T08:19:31.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.214+0000 7f02b3692700 1 Processor -- start 2023-11-08T08:19:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.214+0000 7f02b3692700 1 -- start start 2023-11-08T08:19:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b3692700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f02ac0b1650 0x7f02ac0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b3692700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 0x7f02ac15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b3692700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac159ec0 con 0x7f02ac1599b0 2023-11-08T08:19:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b3692700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac0b1f60 con 0x7f02ac0b1650 2023-11-08T08:19:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b3692700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac15d8a0 con 0x7f02ac0af0a0 2023-11-08T08:19:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b0c2d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f02ac1599b0 0x7f02ac15d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56410/0 (socket says 172.21.15.116:56410) 2023-11-08T08:19:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.215+0000 7f02b0c2d700 1 -- 172.21.15.116:0/1787842985 learned_addr learned my addr 172.21.15.116:0/1787842985 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031476416 0 0) 0x7f02ac159ec0 con 0x7f02ac1599b0 2023-11-08T08:19:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b1c2f700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 0x7f02ac15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 --2- 172.21.15.116:0/1787842985 >> v2:172.21.15.129:3300/0 conn(0x7f02ac0b1650 0x7f02ac0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02980035b0 con 0x7f02ac1599b0 2023-11-08T08:19:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075990638 0 0) 0x7f02980035b0 con 0x7f02ac1599b0 2023-11-08T08:19:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 msgr2=0x7f02ac15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 0x7f02ac15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 >> v2:172.21.15.129:3300/0 conn(0x7f02ac0b1650 msgr2=0x7f02ac0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 --2- 172.21.15.116:0/1787842985 >> v2:172.21.15.129:3300/0 conn(0x7f02ac0b1650 0x7f02ac0b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac15db80 con 0x7f02ac1599b0 2023-11-08T08:19:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f02a8001670 con 0x7f02ac1599b0 2023-11-08T08:19:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 >> v1:172.21.15.116:6789/0 conn(0x7f02ac1599b0 legacy=0x7f02ac15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298003a00 0x7f0298003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 0x7f0298004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0298004d90 0x7f0298005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0b1f60 con 0x7f0298003a00 2023-11-08T08:19:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac15d8a0 con 0x7f0298004d90 2023-11-08T08:19:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029800ba10 con 0x7f02ac0af0a0 2023-11-08T08:19:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298003a00 0x7f0298003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b0c2d700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 0x7f0298004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b1c2f700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0298004d90 0x7f0298005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 -- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 msgr2=0x7f0298004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f02ac0af0a0 0x7f0298004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 -- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0298004d90 msgr2=0x7f0298005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0298004d90 0x7f0298005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b142e700 1 -- 172.21.15.116:0/1787842985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02980035b0 con 0x7f0298003a00 2023-11-08T08:19:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.216+0000 7f02b1c2f700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0298004d90 0x7f0298005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b142e700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298003a00 0x7f0298003e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f029c009b60 tx=0x7f0298005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2746d0e2175cb2ea server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c00e660 con 0x7f0298003a00 2023-11-08T08:19:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c00e7c0 con 0x7f0298003a00 2023-11-08T08:19:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02a3fff700 1 -- 172.21.15.116:0/1787842985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c00eb30 con 0x7f0298003a00 2023-11-08T08:19:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b3692700 1 -- 172.21.15.116:0/1787842985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298003a00 msgr2=0x7f0298003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b3692700 1 --2- 172.21.15.116:0/1787842985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298003a00 0x7f0298003e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f029c009b60 tx=0x7f0298005880 comp rx=0 tx=0).stop 2023-11-08T08:19:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b1c2f700 1 -- 172.21.15.116:0/1787842985 reap_dead start 2023-11-08T08:19:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b3692700 1 -- 172.21.15.116:0/1787842985 shutdown_connections 2023-11-08T08:19:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b3692700 1 -- 172.21.15.116:0/1787842985 >> 172.21.15.116:0/1787842985 conn(0x7f02ac06c010 msgr2=0x7f02ac0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b3692700 1 -- 172.21.15.116:0/1787842985 shutdown_connections 2023-11-08T08:19:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.217+0000 7f02b3692700 1 -- 172.21.15.116:0/1787842985 wait complete. 2023-11-08T08:19:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.219+0000 7f02b3692700 1 Processor -- start 2023-11-08T08:19:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.219+0000 7f02b3692700 1 -- start start 2023-11-08T08:19:31.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0298003a00 0x7f02ac14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 0x7f02ac1e3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02ac1e43d0 0x7f02ac06a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1e4cd0 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1e4fb0 con 0x7f02ac1e43d0 2023-11-08T08:19:31.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1e2ff0 con 0x7f0298003a00 2023-11-08T08:19:31.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 0x7f02ac1e3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 0x7f02ac1e3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36020/0 (socket says 172.21.15.116:36020) 2023-11-08T08:19:31.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 -- 172.21.15.116:0/1540827938 learned_addr learned my addr 172.21.15.116:0/1540827938 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:31.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b142e700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0298003a00 0x7f02ac14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b1c2f700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02ac1e43d0 0x7f02ac06a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 -- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0298003a00 msgr2=0x7f02ac14f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0298003a00 0x7f02ac14f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 -- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02ac1e43d0 msgr2=0x7f02ac06a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02ac1e43d0 0x7f02ac06a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 -- 172.21.15.116:0/1540827938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac15a350 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b0c2d700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 0x7f02ac1e3ad0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f02a8007310 tx=0x7f02a8007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8011580 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80116e0 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8011a90 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.220+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac15de60 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.221+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac15a4e0 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.221+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac1e2e50 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.222+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f02a8011bf0 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.222+0000 7f02a1ffb700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f02900682c0 0x7f029006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.222+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f02a8085e10 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.222+0000 7f02b142e700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f02900682c0 0x7f029006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.223+0000 7f02b142e700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f02900682c0 0x7f029006a770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f029c0090b0 tx=0x7f029c01f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.226+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02ac1e2e50 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.407+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f02ac1e2e50 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.407+0000 7f02a1ffb700 1 -- 172.21.15.116:0/1540827938 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f02ac1e2e50 con 0x7f02ac0b4cb0 2023-11-08T08:19:31.409 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.409+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f02900682c0 msgr2=0x7f029006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.409+0000 7f02b3692700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f02900682c0 0x7f029006a770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f029c0090b0 tx=0x7f029c01f040 comp rx=0 tx=0).stop 2023-11-08T08:19:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.409+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 msgr2=0x7f02ac1e3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.409+0000 7f02b3692700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 0x7f02ac1e3ad0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f02a8007310 tx=0x7f02a8007620 comp rx=0 tx=0).stop 2023-11-08T08:19:31.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.409+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 shutdown_connections 2023-11-08T08:19:31.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.409+0000 7f02b3692700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f02900682c0 0x7f029006a770 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.410+0000 7f02b3692700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02ac1e43d0 0x7f02ac06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.410+0000 7f02b3692700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0298003a00 0x7f02ac14f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.410+0000 7f02b3692700 1 --2- 172.21.15.116:0/1540827938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ac0b4cb0 0x7f02ac1e3ad0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.410+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 >> 172.21.15.116:0/1540827938 conn(0x7f02ac06c010 msgr2=0x7f02ac0b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:31.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.410+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 shutdown_connections 2023-11-08T08:19:31.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.410+0000 7f02b3692700 1 -- 172.21.15.116:0/1540827938 wait complete. 2023-11-08T08:19:31.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 Processor -- start 2023-11-08T08:19:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 -- start start 2023-11-08T08:19:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde28160300 0x7fde2815d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde280cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2815ceb0 con 0x7fde280cec00 2023-11-08T08:19:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde281608c0 con 0x7fde28160300 2023-11-08T08:19:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde2e189700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde28160ba0 con 0x7fde2815d880 2023-11-08T08:19:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.546+0000 7fde26ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde28160300 0x7fde2815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde28160300 0x7fde2815d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36028/0 (socket says 172.21.15.116:36028) 2023-11-08T08:19:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 -- 172.21.15.116:0/769878503 learned_addr learned my addr 172.21.15.116:0/769878503 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde27fff700 1 --2- 172.21.15.116:0/769878503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde280cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 -- 172.21.15.116:0/769878503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 msgr2=0x7fde280cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 --2- 172.21.15.116:0/769878503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde280cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 -- 172.21.15.116:0/769878503 >> v1:172.21.15.129:6789/0 conn(0x7fde280cec00 legacy=0x7fde28065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 -- 172.21.15.116:0/769878503 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28160e80 con 0x7fde28160300 2023-11-08T08:19:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde26ffd700 1 --2- 172.21.15.116:0/769878503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde28160300 0x7fde2815d340 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0098f0 tx=0x7fde1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f97cff4d175a40 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde267fc700 1 -- 172.21.15.116:0/769878503 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c004230 con 0x7fde28160300 2023-11-08T08:19:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde267fc700 1 -- 172.21.15.116:0/769878503 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c004390 con 0x7fde28160300 2023-11-08T08:19:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.547+0000 7fde267fc700 1 -- 172.21.15.116:0/769878503 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0036a0 con 0x7fde28160300 2023-11-08T08:19:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 -- 172.21.15.116:0/769878503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde28160300 msgr2=0x7fde2815d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 --2- 172.21.15.116:0/769878503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde28160300 0x7fde2815d340 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0098f0 tx=0x7fde1c009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 -- 172.21.15.116:0/769878503 shutdown_connections 2023-11-08T08:19:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 --2- 172.21.15.116:0/769878503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde280cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 --2- 172.21.15.116:0/769878503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde28160300 0x7fde2815d340 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 -- 172.21.15.116:0/769878503 >> 172.21.15.116:0/769878503 conn(0x7fde280cf400 msgr2=0x7fde280cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 -- 172.21.15.116:0/769878503 shutdown_connections 2023-11-08T08:19:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.548+0000 7fde2e189700 1 -- 172.21.15.116:0/769878503 wait complete. 2023-11-08T08:19:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.549+0000 7fde2e189700 1 Processor -- start 2023-11-08T08:19:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.549+0000 7fde2e189700 1 -- start start 2023-11-08T08:19:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 0x7fde2815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde28071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde28160300 0x7fde28070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde281535f0 con 0x7fde280cec00 2023-11-08T08:19:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde281538d0 con 0x7fde28160300 2023-11-08T08:19:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde28153bb0 con 0x7fde2815d880 2023-11-08T08:19:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 0x7fde2815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 0x7fde2815e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36038/0 (socket says 172.21.15.116:36038) 2023-11-08T08:19:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 -- 172.21.15.116:0/1716127651 learned_addr learned my addr 172.21.15.116:0/1716127651 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde26ffd700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde28071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde27fff700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde28160300 0x7fde28070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 -- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 msgr2=0x7fde28071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde28071ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 -- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde28160300 msgr2=0x7fde28070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde28160300 0x7fde28070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 -- 172.21.15.116:0/1716127651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c009550 con 0x7fde280cec00 2023-11-08T08:19:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde277fe700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 0x7fde2815e0b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fde1800b4a0 tx=0x7fde1800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18011830 con 0x7fde280cec00 2023-11-08T08:19:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18011990 con 0x7fde280cec00 2023-11-08T08:19:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde280cefd0 con 0x7fde280cec00 2023-11-08T08:19:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18011d40 con 0x7fde280cec00 2023-11-08T08:19:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.550+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde28161160 con 0x7fde280cec00 2023-11-08T08:19:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.552+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde1800f050 con 0x7fde280cec00 2023-11-08T08:19:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.552+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde280cf160 con 0x7fde280cec00 2023-11-08T08:19:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.552+0000 7fde0ffff700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fde140682f0 0x7fde1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.552+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fde18085870 con 0x7fde280cec00 2023-11-08T08:19:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.552+0000 7fde26ffd700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fde140682f0 0x7fde1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.552+0000 7fde26ffd700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fde140682f0 0x7fde1406a7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fde281f5ee0 tx=0x7fde1c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.556+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde18052070 con 0x7fde280cec00 2023-11-08T08:19:31.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.734+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde280743f0 con 0x7fde280cec00 2023-11-08T08:19:31.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.735+0000 7fde0ffff700 1 -- 172.21.15.116:0/1716127651 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde18055690 con 0x7fde280cec00 2023-11-08T08:19:31.736 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:31.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fde140682f0 msgr2=0x7fde1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fde140682f0 0x7fde1406a7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fde281f5ee0 tx=0x7fde1c008030 comp rx=0 tx=0).stop 2023-11-08T08:19:31.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 msgr2=0x7fde2815e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 0x7fde2815e0b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fde1800b4a0 tx=0x7fde1800b860 comp rx=0 tx=0).stop 2023-11-08T08:19:31.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 shutdown_connections 2023-11-08T08:19:31.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fde140682f0 0x7fde1406a7a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde28160300 0x7fde28070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde280cec00 0x7fde2815e0b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 --2- 172.21.15.116:0/1716127651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fde2815d880 0x7fde28071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 >> 172.21.15.116:0/1716127651 conn(0x7fde280cf400 msgr2=0x7fde2806ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:31.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 shutdown_connections 2023-11-08T08:19:31.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.737+0000 7fde2e189700 1 -- 172.21.15.116:0/1716127651 wait complete. 2023-11-08T08:19:31.750 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:19:31.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.869+0000 7fe06b1c1700 1 Processor -- start 2023-11-08T08:19:31.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.869+0000 7fe06b1c1700 1 -- start start 2023-11-08T08:19:31.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.869+0000 7fe06b1c1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe06406a270 0x7fe06406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.869+0000 7fe06b1c1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe06415cc80 0x7fe06406de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe06b1c1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06406ab80 con 0x7fe06406e3a0 2023-11-08T08:19:31.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe06b1c1700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0640b2490 con 0x7fe06415cc80 2023-11-08T08:19:31.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe06b1c1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0640b1810 con 0x7fe06406a270 2023-11-08T08:19:31.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe06415cc80 0x7fe06406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe06415cc80 0x7fe06406de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36042/0 (socket says 172.21.15.116:36042) 2023-11-08T08:19:31.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 -- 172.21.15.116:0/2028171786 learned_addr learned my addr 172.21.15.116:0/2028171786 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:31.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe068f5d700 1 --2- 172.21.15.116:0/2028171786 >> v2:172.21.15.129:3300/0 conn(0x7fe06406a270 0x7fe06406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 -- 172.21.15.116:0/2028171786 >> v2:172.21.15.129:3300/0 conn(0x7fe06406a270 msgr2=0x7fe06406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:31.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 --2- 172.21.15.116:0/2028171786 >> v2:172.21.15.129:3300/0 conn(0x7fe06406a270 0x7fe06406a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 -- 172.21.15.116:0/2028171786 >> v1:172.21.15.129:6789/0 conn(0x7fe06406e3a0 legacy=0x7fe06406ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 -- 172.21.15.116:0/2028171786 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640b1af0 con 0x7fe06415cc80 2023-11-08T08:19:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe063fff700 1 --2- 172.21.15.116:0/2028171786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406de60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe064067380 tx=0x7fe050009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4695c82b0eff3e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe0637fe700 1 -- 172.21.15.116:0/2028171786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050011830 con 0x7fe06415cc80 2023-11-08T08:19:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe0637fe700 1 -- 172.21.15.116:0/2028171786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe050011990 con 0x7fe06415cc80 2023-11-08T08:19:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.870+0000 7fe0637fe700 1 -- 172.21.15.116:0/2028171786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050011d20 con 0x7fe06415cc80 2023-11-08T08:19:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 -- 172.21.15.116:0/2028171786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 msgr2=0x7fe06406de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/2028171786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406de60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe064067380 tx=0x7fe050009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 -- 172.21.15.116:0/2028171786 shutdown_connections 2023-11-08T08:19:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/2028171786 >> v2:172.21.15.129:3300/0 conn(0x7fe06406a270 0x7fe06406a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/2028171786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406de60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 -- 172.21.15.116:0/2028171786 >> 172.21.15.116:0/2028171786 conn(0x7fe06406c010 msgr2=0x7fe0640cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 -- 172.21.15.116:0/2028171786 shutdown_connections 2023-11-08T08:19:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.871+0000 7fe06b1c1700 1 -- 172.21.15.116:0/2028171786 wait complete. 2023-11-08T08:19:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.872+0000 7fe06b1c1700 1 Processor -- start 2023-11-08T08:19:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.872+0000 7fe06b1c1700 1 -- start start 2023-11-08T08:19:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06b1c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe06406a270 0x7fe064145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06b1c1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe06406e3a0 0x7fe0640b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06b1c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06b1c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064146500 con 0x7fe06415cc80 2023-11-08T08:19:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06b1c1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0640b4ce0 con 0x7fe06406e3a0 2023-11-08T08:19:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06b1c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0640b4fc0 con 0x7fe06406a270 2023-11-08T08:19:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36046/0 (socket says 172.21.15.116:36046) 2023-11-08T08:19:31.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 -- 172.21.15.116:0/3351341551 learned_addr learned my addr 172.21.15.116:0/3351341551 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:31.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe063fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe06406e3a0 0x7fe0640b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe068f5d700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe06406a270 0x7fe064145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 -- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe06406a270 msgr2=0x7fe064145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe06406a270 0x7fe064145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 -- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe06406e3a0 msgr2=0x7fe0640b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe06406e3a0 0x7fe0640b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.873+0000 7fe06975e700 1 -- 172.21.15.116:0/3351341551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050009550 con 0x7fe06415cc80 2023-11-08T08:19:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.874+0000 7fe06975e700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406ce20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00cb50 tx=0x7fe05c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.874+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c003c00 con 0x7fe06415cc80 2023-11-08T08:19:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.874+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c003d60 con 0x7fe06415cc80 2023-11-08T08:19:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.874+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0640b20b0 con 0x7fe06415cc80 2023-11-08T08:19:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.874+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c00aa20 con 0x7fe06415cc80 2023-11-08T08:19:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.874+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0640b1dd0 con 0x7fe06415cc80 2023-11-08T08:19:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.875+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06406e7c0 con 0x7fe06415cc80 2023-11-08T08:19:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.875+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe05c010050 con 0x7fe06415cc80 2023-11-08T08:19:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.875+0000 7fe0617fa700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0480682f0 0x7fe04806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.876+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe05c085b20 con 0x7fe06415cc80 2023-11-08T08:19:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.876+0000 7fe068f5d700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0480682f0 0x7fe04806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.876+0000 7fe068f5d700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0480682f0 0x7fe04806a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe058009b40 tx=0x7fe058009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:31.880+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe05c052380 con 0x7fe06415cc80 2023-11-08T08:19:32.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.100+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0640743f0 con 0x7fe06415cc80 2023-11-08T08:19:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.101+0000 7fe0617fa700 1 -- 172.21.15.116:0/3351341551 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4654 (secure 0 0 0) 0x7fe05c0559a0 con 0x7fe06415cc80 2023-11-08T08:19:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:32.102 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:18:42.885473+0000","modified":"2023-11-08T08:18:45.898738+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-08T08:19:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0480682f0 msgr2=0x7fe04806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0480682f0 0x7fe04806a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe058009b40 tx=0x7fe058009210 comp rx=0 tx=0).stop 2023-11-08T08:19:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 msgr2=0x7fe06406ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406ce20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00cb50 tx=0x7fe05c00ce60 comp rx=0 tx=0).stop 2023-11-08T08:19:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 shutdown_connections 2023-11-08T08:19:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06415cc80 0x7fe06406ce20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe06406a270 0x7fe064145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0480682f0 0x7fe04806a7a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 --2- 172.21.15.116:0/3351341551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe06406e3a0 0x7fe0640b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 >> 172.21.15.116:0/3351341551 conn(0x7fe06406c010 msgr2=0x7fe0640b0140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 shutdown_connections 2023-11-08T08:19:32.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.103+0000 7fe06b1c1700 1 -- 172.21.15.116:0/3351341551 wait complete. 2023-11-08T08:19:32.107 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-08T08:19:32.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.234+0000 7f348351f700 1 Processor -- start 2023-11-08T08:19:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.234+0000 7f348351f700 1 -- start start 2023-11-08T08:19:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f348351f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c06e400 0x7f347c0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f348351f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f347c160e70 0x7f347c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f348351f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c06e900 con 0x7f347c0d0c60 2023-11-08T08:19:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f348351f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c160390 con 0x7f347c160e70 2023-11-08T08:19:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f348351f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c160670 con 0x7f347c06e400 2023-11-08T08:19:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f34812bb700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f347c0d0c60 0x7f347c0d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56426/0 (socket says 172.21.15.116:56426) 2023-11-08T08:19:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f34812bb700 1 -- 172.21.15.116:0/939049656 learned_addr learned my addr 172.21.15.116:0/939049656 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3481abc700 1 --2- 172.21.15.116:0/939049656 >> v2:172.21.15.116:3300/0 conn(0x7f347c160e70 0x7f347c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3480aba700 1 --2- 172.21.15.116:0/939049656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c06e400 0x7f347c0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3473fff700 1 -- 172.21.15.116:0/939049656 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963687507 0 0) 0x7f347c06e900 con 0x7f347c0d0c60 2023-11-08T08:19:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3473fff700 1 -- 172.21.15.116:0/939049656 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34680035b0 con 0x7f347c0d0c60 2023-11-08T08:19:32.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3481abc700 1 -- 172.21.15.116:0/939049656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c06e400 msgr2=0x7f347c0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:32.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3481abc700 1 --2- 172.21.15.116:0/939049656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c06e400 0x7f347c0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3481abc700 1 -- 172.21.15.116:0/939049656 >> v1:172.21.15.116:6789/0 conn(0x7f347c0d0c60 legacy=0x7f347c0d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.235+0000 7f3481abc700 1 -- 172.21.15.116:0/939049656 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c160950 con 0x7f347c160e70 2023-11-08T08:19:32.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f3481abc700 1 --2- 172.21.15.116:0/939049656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15c970 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3474009d90 tx=0x7f347400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456c330cef67b691 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f3473fff700 1 -- 172.21.15.116:0/939049656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400e070 con 0x7f347c160e70 2023-11-08T08:19:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f3473fff700 1 -- 172.21.15.116:0/939049656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400bb20 con 0x7f347c160e70 2023-11-08T08:19:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f3473fff700 1 -- 172.21.15.116:0/939049656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400be90 con 0x7f347c160e70 2023-11-08T08:19:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 -- 172.21.15.116:0/939049656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 msgr2=0x7f347c15c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 --2- 172.21.15.116:0/939049656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15c970 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3474009d90 tx=0x7f347400b330 comp rx=0 tx=0).stop 2023-11-08T08:19:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 -- 172.21.15.116:0/939049656 shutdown_connections 2023-11-08T08:19:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 --2- 172.21.15.116:0/939049656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15c970 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 --2- 172.21.15.116:0/939049656 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c06e400 0x7f347c0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 -- 172.21.15.116:0/939049656 >> 172.21.15.116:0/939049656 conn(0x7f347c06a270 msgr2=0x7f347c06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 -- 172.21.15.116:0/939049656 shutdown_connections 2023-11-08T08:19:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.236+0000 7f348351f700 1 -- 172.21.15.116:0/939049656 wait complete. 2023-11-08T08:19:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 Processor -- start 2023-11-08T08:19:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 -- start start 2023-11-08T08:19:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06e400 0x7f347c14e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c0d0c60 0x7f347c15e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c15f9b0 con 0x7f347c160e70 2023-11-08T08:19:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0b17c0 con 0x7f347c06e400 2023-11-08T08:19:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f348351f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c153460 con 0x7f347c0d0c60 2023-11-08T08:19:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f3481abc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f3481abc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36068/0 (socket says 172.21.15.116:36068) 2023-11-08T08:19:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f3481abc700 1 -- 172.21.15.116:0/2316205472 learned_addr learned my addr 172.21.15.116:0/2316205472 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f34812bb700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06e400 0x7f347c14e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.238+0000 7f3480aba700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c0d0c60 0x7f347c15e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3481abc700 1 -- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c0d0c60 msgr2=0x7f347c15e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3481abc700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c0d0c60 0x7f347c15e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3481abc700 1 -- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06e400 msgr2=0x7f347c14e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3481abc700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06e400 0x7f347c14e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3481abc700 1 -- 172.21.15.116:0/2316205472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474009a70 con 0x7f347c160e70 2023-11-08T08:19:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3481abc700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15e9d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3474014040 tx=0x7f3474004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474004800 con 0x7f347c160e70 2023-11-08T08:19:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3474020070 con 0x7f347c160e70 2023-11-08T08:19:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c15ceb0 con 0x7f347c160e70 2023-11-08T08:19:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34740276c0 con 0x7f347c160e70 2023-11-08T08:19:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.239+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c0d0740 con 0x7f347c160e70 2023-11-08T08:19:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.240+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c15d040 con 0x7f347c160e70 2023-11-08T08:19:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.241+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f347402e070 con 0x7f347c160e70 2023-11-08T08:19:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.241+0000 7f3471ffb700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34600682f0 0x7f346006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.241+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3474022070 con 0x7f347c160e70 2023-11-08T08:19:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.241+0000 7f34812bb700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34600682f0 0x7f346006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.242+0000 7f34812bb700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34600682f0 0x7f346006a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f346c007fd0 tx=0x7f346c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.246+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3474054050 con 0x7f347c160e70 2023-11-08T08:19:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.429+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f347c0743f0 con 0x7f347c160e70 2023-11-08T08:19:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.430+0000 7f3471ffb700 1 -- 172.21.15.116:0/2316205472 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f3474057670 con 0x7f347c160e70 2023-11-08T08:19:32.432 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:32.432 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34600682f0 msgr2=0x7f346006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34600682f0 0x7f346006a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f346c007fd0 tx=0x7f346c009040 comp rx=0 tx=0).stop 2023-11-08T08:19:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 msgr2=0x7f347c15e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15e9d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3474014040 tx=0x7f3474004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 shutdown_connections 2023-11-08T08:19:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f347c160e70 0x7f347c15e9d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f34600682f0 0x7f346006a7a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c06e400 0x7f347c14e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 --2- 172.21.15.116:0/2316205472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f347c0d0c60 0x7f347c15e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.432+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 >> 172.21.15.116:0/2316205472 conn(0x7f347c06a270 msgr2=0x7f347c06b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.433+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 shutdown_connections 2023-11-08T08:19:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.433+0000 7f348351f700 1 -- 172.21.15.116:0/2316205472 wait complete. 2023-11-08T08:19:32.446 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-08T08:19:32.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:19:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.566+0000 7f0dd7d53700 1 Processor -- start 2023-11-08T08:19:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.566+0000 7f0dd7d53700 1 -- start start 2023-11-08T08:19:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd7d53700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0dd006a270 0x7f0dd006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd7d53700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0dd015cc80 0x7f0dd006de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd7d53700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006e3a0 0x7f0dd006ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd7d53700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd00b2490 con 0x7f0dd015cc80 2023-11-08T08:19:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd7d53700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd006ab80 con 0x7f0dd006a270 2023-11-08T08:19:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd7d53700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b1810 con 0x7f0dd006e3a0 2023-11-08T08:19:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0dd015cc80 0x7f0dd006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0dd015cc80 0x7f0dd006de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38022/0 (socket says 172.21.15.116:38022) 2023-11-08T08:19:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 -- 172.21.15.116:0/1355982359 learned_addr learned my addr 172.21.15.116:0/1355982359 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd5aef700 1 --2- 172.21.15.116:0/1355982359 >> v2:172.21.15.129:3300/0 conn(0x7f0dd006a270 0x7f0dd006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd62f0700 1 --2- 172.21.15.116:0/1355982359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006e3a0 0x7f0dd006ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 -- 172.21.15.116:0/1355982359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006e3a0 msgr2=0x7f0dd006ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 --2- 172.21.15.116:0/1355982359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006e3a0 0x7f0dd006ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 -- 172.21.15.116:0/1355982359 >> v2:172.21.15.129:3300/0 conn(0x7f0dd006a270 msgr2=0x7f0dd006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 --2- 172.21.15.116:0/1355982359 >> v2:172.21.15.129:3300/0 conn(0x7f0dd006a270 0x7f0dd006a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 -- 172.21.15.116:0/1355982359 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00b1af0 con 0x7f0dd015cc80 2023-11-08T08:19:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd5aef700 1 --2- 172.21.15.116:0/1355982359 >> v2:172.21.15.129:3300/0 conn(0x7f0dd006a270 0x7f0dd006a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd52ee700 1 --2- 172.21.15.116:0/1355982359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd006de60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0067380 tx=0x7f0dc0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677bb8a5b941c2bb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd4aed700 1 -- 172.21.15.116:0/1355982359 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc000b870 con 0x7f0dd015cc80 2023-11-08T08:19:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd4aed700 1 -- 172.21.15.116:0/1355982359 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc000b9d0 con 0x7f0dd015cc80 2023-11-08T08:19:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.567+0000 7f0dd4aed700 1 -- 172.21.15.116:0/1355982359 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc000bd60 con 0x7f0dd015cc80 2023-11-08T08:19:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 -- 172.21.15.116:0/1355982359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 msgr2=0x7f0dd006de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/1355982359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd006de60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0067380 tx=0x7f0dc0009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 -- 172.21.15.116:0/1355982359 shutdown_connections 2023-11-08T08:19:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/1355982359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006e3a0 0x7f0dd006ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/1355982359 >> v2:172.21.15.129:3300/0 conn(0x7f0dd006a270 0x7f0dd006a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/1355982359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd006de60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 -- 172.21.15.116:0/1355982359 >> 172.21.15.116:0/1355982359 conn(0x7f0dd006c010 msgr2=0x7f0dd00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 -- 172.21.15.116:0/1355982359 shutdown_connections 2023-11-08T08:19:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.568+0000 7f0dd7d53700 1 -- 172.21.15.116:0/1355982359 wait complete. 2023-11-08T08:19:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.569+0000 7f0dd7d53700 1 Processor -- start 2023-11-08T08:19:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.569+0000 7f0dd7d53700 1 -- start start 2023-11-08T08:19:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd7d53700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006a270 0x7f0dd01533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd7d53700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dd006e3a0 0x7f0dd00b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd7d53700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd7d53700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00c3bc0 con 0x7f0dd015cc80 2023-11-08T08:19:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd7d53700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01f3320 con 0x7f0dd006e3a0 2023-11-08T08:19:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd7d53700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b0bf0 con 0x7f0dd006a270 2023-11-08T08:19:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd015b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38024/0 (socket says 172.21.15.116:38024) 2023-11-08T08:19:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 -- 172.21.15.116:0/3092606290 learned_addr learned my addr 172.21.15.116:0/3092606290 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd52ee700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dd006e3a0 0x7f0dd00b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd5aef700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006a270 0x7f0dd01533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 -- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006a270 msgr2=0x7f0dd01533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006a270 0x7f0dd01533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 -- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dd006e3a0 msgr2=0x7f0dd00b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dd006e3a0 0x7f0dd00b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd62f0700 1 -- 172.21.15.116:0/3092606290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0009550 con 0x7f0dd015cc80 2023-11-08T08:19:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.570+0000 7f0dd5aef700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006a270 0x7f0dd01533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.571+0000 7f0dd62f0700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd015b880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8007e60 tx=0x7f0dc800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.571+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800ede0 con 0x7f0dd015cc80 2023-11-08T08:19:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.571+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc8014420 con 0x7f0dd015cc80 2023-11-08T08:19:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.571+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd00b20b0 con 0x7f0dd015cc80 2023-11-08T08:19:32.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.571+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80147d0 con 0x7f0dd015cc80 2023-11-08T08:19:32.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.571+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd00b1dd0 con 0x7f0dd015cc80 2023-11-08T08:19:32.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.572+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd006e800 con 0x7f0dd015cc80 2023-11-08T08:19:32.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.572+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0dc8012050 con 0x7f0dd015cc80 2023-11-08T08:19:32.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.572+0000 7f0dc67fc700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0db40682f0 0x7f0db406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.573+0000 7f0dd5aef700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0db40682f0 0x7f0db406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.573+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0dc80862b0 con 0x7f0dd015cc80 2023-11-08T08:19:32.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.573+0000 7f0dd5aef700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0db40682f0 0x7f0db406a7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009820 tx=0x7f0dcc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.577+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0dc8052b10 con 0x7f0dd015cc80 2023-11-08T08:19:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.796+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dd01f3460 con 0x7f0dd015cc80 2023-11-08T08:19:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.797+0000 7f0dc67fc700 1 -- 172.21.15.116:0/3092606290 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4654 (secure 0 0 0) 0x7f0dd01f3460 con 0x7f0dd015cc80 2023-11-08T08:19:32.798 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:32.799 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:18:42.885473+0000","modified":"2023-11-08T08:18:45.898738+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-08T08:19:32.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0db40682f0 msgr2=0x7f0db406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0db40682f0 0x7f0db406a7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009820 tx=0x7f0dcc008040 comp rx=0 tx=0).stop 2023-11-08T08:19:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 msgr2=0x7f0dd015b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd015b880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8007e60 tx=0x7f0dc800e3c0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 shutdown_connections 2023-11-08T08:19:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd015cc80 0x7f0dd015b880 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0dd006a270 0x7f0dd01533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0db40682f0 0x7f0db406a7a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 --2- 172.21.15.116:0/3092606290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dd006e3a0 0x7f0dd00b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 >> 172.21.15.116:0/3092606290 conn(0x7f0dd006c010 msgr2=0x7f0dd00ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:32.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 shutdown_connections 2023-11-08T08:19:32.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.799+0000 7f0dd7d53700 1 -- 172.21.15.116:0/3092606290 wait complete. 2023-11-08T08:19:32.803 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-08T08:19:32.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:19:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.931+0000 7f6ee45e4700 1 Processor -- start 2023-11-08T08:19:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.931+0000 7f6ee45e4700 1 -- start start 2023-11-08T08:19:32.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.932+0000 7f6ee45e4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 0x7f6edc0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.932+0000 7f6ee45e4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc0b4fc0 con 0x7f6edc0ce1e0 2023-11-08T08:19:32.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.932+0000 7f6ee45e4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc0b05e0 con 0x7f6edc06feb0 2023-11-08T08:19:32.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.932+0000 7f6ee45e4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc159c30 con 0x7f6edc0ced70 2023-11-08T08:19:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.932+0000 7f6ee2b81700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6edc0ce1e0 0x7f6edc06a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48708/0 (socket says 172.21.15.116:48708) 2023-11-08T08:19:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.932+0000 7f6ee2b81700 1 -- 172.21.15.116:0/676046019 learned_addr learned my addr 172.21.15.116:0/676046019 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949589423 0 0) 0x7f6edc0b4fc0 con 0x7f6edc0ce1e0 2023-11-08T08:19:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec80035b0 con 0x7f6edc0ce1e0 2023-11-08T08:19:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2380700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 0x7f6edc0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701018652 0 0) 0x7f6ec80035b0 con 0x7f6edc0ce1e0 2023-11-08T08:19:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 msgr2=0x7f6edc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 0x7f6edc0b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 >> v1:172.21.15.129:6789/0 conn(0x7f6edc06feb0 legacy=0x7f6edc06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0b08c0 con 0x7f6edc0ce1e0 2023-11-08T08:19:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f6ed40019f0 con 0x7f6edc0ce1e0 2023-11-08T08:19:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 >> v1:172.21.15.116:6789/0 conn(0x7f6edc0ce1e0 legacy=0x7f6edc06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 0x7f6ec8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec8003fb0 0x7f6ec8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6ec8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc159c30 con 0x7f6ec8004d90 2023-11-08T08:19:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec800baf0 con 0x7f6ec8003fb0 2023-11-08T08:19:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec800dec0 con 0x7f6edc0ced70 2023-11-08T08:19:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2b81700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6ec8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee1b7f700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec8003fb0 0x7f6ec8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2380700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 0x7f6ec8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2b81700 1 -- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 msgr2=0x7f6ec8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:32.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2b81700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0ced70 0x7f6ec8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2b81700 1 -- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec8003fb0 msgr2=0x7f6ec8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:32.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2b81700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec8003fb0 0x7f6ec8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.933+0000 7f6ee2b81700 1 -- 172.21.15.116:0/676046019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80035b0 con 0x7f6ec8004d90 2023-11-08T08:19:32.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee2b81700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6ec8005200 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4002230 tx=0x7f6ec8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1531971a80d21fa server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4012600 con 0x7f6ec8004d90 2023-11-08T08:19:32.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed40127a0 con 0x7f6ec8004d90 2023-11-08T08:19:32.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee137e700 1 -- 172.21.15.116:0/676046019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4012b50 con 0x7f6ec8004d90 2023-11-08T08:19:32.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee45e4700 1 -- 172.21.15.116:0/676046019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 msgr2=0x7f6ec8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/676046019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6ec8005200 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4002230 tx=0x7f6ec8005880 comp rx=0 tx=0).stop 2023-11-08T08:19:32.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee2b81700 1 -- 172.21.15.116:0/676046019 reap_dead start 2023-11-08T08:19:32.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee45e4700 1 -- 172.21.15.116:0/676046019 shutdown_connections 2023-11-08T08:19:32.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee45e4700 1 -- 172.21.15.116:0/676046019 >> 172.21.15.116:0/676046019 conn(0x7f6edc0b2fc0 msgr2=0x7f6edc071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:32.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee45e4700 1 -- 172.21.15.116:0/676046019 shutdown_connections 2023-11-08T08:19:32.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.934+0000 7f6ee45e4700 1 -- 172.21.15.116:0/676046019 wait complete. 2023-11-08T08:19:32.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.936+0000 7f6ee45e4700 1 Processor -- start 2023-11-08T08:19:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.936+0000 7f6ee45e4700 1 -- start start 2023-11-08T08:19:32.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6edc0c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0b4900 0x7f6edc0c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc0c5080 0x7f6edc0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0c4bf0 con 0x7f6ec8004d90 2023-11-08T08:19:32.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc1edca0 con 0x7f6edc0c5080 2023-11-08T08:19:32.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0c3fe0 con 0x7f6edc0b4900 2023-11-08T08:19:32.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6edc0c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6edc0c3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38042/0 (socket says 172.21.15.116:38042) 2023-11-08T08:19:32.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 -- 172.21.15.116:0/1523685702 learned_addr learned my addr 172.21.15.116:0/1523685702 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:32.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2b81700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc0c5080 0x7f6edc0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee1b7f700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0b4900 0x7f6edc0c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 -- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0b4900 msgr2=0x7f6edc0c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0b4900 0x7f6edc0c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 -- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc0c5080 msgr2=0x7f6edc0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc0c5080 0x7f6edc0c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 -- 172.21.15.116:0/1523685702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0b5260 con 0x7f6ec8004d90 2023-11-08T08:19:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2b81700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc0c5080 0x7f6edc0c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee2380700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6edc0c3970 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8011aa0 tx=0x7f6ed8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800f980 con 0x7f6ec8004d90 2023-11-08T08:19:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed800fae0 con 0x7f6ec8004d90 2023-11-08T08:19:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc0b0ba0 con 0x7f6ec8004d90 2023-11-08T08:19:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.937+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800fe90 con 0x7f6ec8004d90 2023-11-08T08:19:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.938+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc072420 con 0x7f6ec8004d90 2023-11-08T08:19:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.938+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc0c4ed0 con 0x7f6ec8004d90 2023-11-08T08:19:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.939+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ed8014440 con 0x7f6ec8004d90 2023-11-08T08:19:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.939+0000 7f6ed2ffd700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ec00682f0 0x7f6ec006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.939+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6ed8086570 con 0x7f6ec8004d90 2023-11-08T08:19:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.939+0000 7f6ee1b7f700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ec00682f0 0x7f6ec006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.940+0000 7f6ee1b7f700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ec00682f0 0x7f6ec006a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0b1510 tx=0x7f6ecc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:32.943+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ed8052e30 con 0x7f6ec8004d90 2023-11-08T08:19:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.164+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6edc067370 con 0x7f6ec8004d90 2023-11-08T08:19:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.164+0000 7f6ed2ffd700 1 -- 172.21.15.116:0/1523685702 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4654 (secure 0 0 0) 0x7f6ed8056450 con 0x7f6ec8004d90 2023-11-08T08:19:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:33.166 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3890421971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3890421971},{"type":"v1","addr":"172.21.15.116:6837","nonce":3890421971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1910300273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1910300273},{"type":"v1","addr":"172.21.15.129:6833","nonce":1910300273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2014055550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2014055550},{"type":"v1","addr":"172.21.15.116:6835","nonce":2014055550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:18:42.885473+0000","modified":"2023-11-08T08:18:45.898738+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1536263974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1536263974},{"type":"v1","addr":"172.21.15.129:6835","nonce":1536263974}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-08T08:19:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ec00682f0 msgr2=0x7f6ec006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ec00682f0 0x7f6ec006a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0b1510 tx=0x7f6ecc009040 comp rx=0 tx=0).stop 2023-11-08T08:19:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 msgr2=0x7f6edc0c3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6edc0c3970 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8011aa0 tx=0x7f6ed8011e60 comp rx=0 tx=0).stop 2023-11-08T08:19:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 shutdown_connections 2023-11-08T08:19:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ec00682f0 0x7f6ec006a7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc0c5080 0x7f6edc0c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec8004d90 0x7f6edc0c3970 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 --2- 172.21.15.116:0/1523685702 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6edc0b4900 0x7f6edc0c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 >> 172.21.15.116:0/1523685702 conn(0x7f6edc0b2fc0 msgr2=0x7f6edc06af40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 shutdown_connections 2023-11-08T08:19:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.166+0000 7f6ee45e4700 1 -- 172.21.15.116:0/1523685702 wait complete. 2023-11-08T08:19:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-08T08:19:33.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 Processor -- start 2023-11-08T08:19:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 -- start start 2023-11-08T08:19:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f52940b4660 0x7f5294071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52940b4a30 0x7f52940b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940cfb00 con 0x7f52940cff90 2023-11-08T08:19:33.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529415aa20 con 0x7f52940b4660 2023-11-08T08:19:33.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f529abf7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52940cf4d0 con 0x7f52940b4a30 2023-11-08T08:19:33.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f5293fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f52940cff90 0x7f529406b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48722/0 (socket says 172.21.15.116:48722) 2023-11-08T08:19:33.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.298+0000 7f5293fff700 1 -- 172.21.15.116:0/2543153433 learned_addr learned my addr 172.21.15.116:0/2543153433 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:33.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 --2- 172.21.15.116:0/2543153433 >> v2:172.21.15.129:3300/0 conn(0x7f52940b4660 0x7f5294071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5299194700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52940b4a30 0x7f52940b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701839877 0 0) 0x7f52940cfb00 con 0x7f52940cff90 2023-11-08T08:19:33.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52840035b0 con 0x7f52940cff90 2023-11-08T08:19:33.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1611820682 0 0) 0x7f52840035b0 con 0x7f52940cff90 2023-11-08T08:19:33.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52940b4a30 msgr2=0x7f52940b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:33.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52940b4a30 0x7f52940b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 >> v2:172.21.15.129:3300/0 conn(0x7f52940b4660 msgr2=0x7f5294071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:33.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 --2- 172.21.15.116:0/2543153433 >> v2:172.21.15.129:3300/0 conn(0x7f52940b4660 0x7f5294071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5294072550 con 0x7f52940cff90 2023-11-08T08:19:33.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f5280001670 con 0x7f52940cff90 2023-11-08T08:19:33.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 --2- 172.21.15.116:0/2543153433 >> v2:172.21.15.129:3300/0 conn(0x7f52940b4660 0x7f5294071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:33.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 >> v1:172.21.15.116:6789/0 conn(0x7f52940cff90 legacy=0x7f529406b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284003a00 0x7f5284003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284004430 0x7f52840048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284004fb0 0x7f528400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529415aa20 con 0x7f5284003a00 2023-11-08T08:19:33.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940cf4d0 con 0x7f5284004fb0 2023-11-08T08:19:33.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528400bf50 con 0x7f5284004430 2023-11-08T08:19:33.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5299194700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52940b4a30 0x7f52940b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:33.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284003a00 0x7f5284003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5293fff700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284004430 0x7f52840048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5299194700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284004fb0 0x7f528400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 -- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284004430 msgr2=0x7f52840048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:33.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284004430 0x7f52840048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 -- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284004fb0 msgr2=0x7f528400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:33.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284004fb0 0x7f528400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.299+0000 7f5298993700 1 -- 172.21.15.116:0/2543153433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52840035b0 con 0x7f5284003a00 2023-11-08T08:19:33.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f5298993700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284003a00 0x7f5284003e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5288009d30 tx=0x7f5284005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568015f90c46244d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:33.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288009e80 con 0x7f5284003a00 2023-11-08T08:19:33.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528800e9a0 con 0x7f5284003a00 2023-11-08T08:19:33.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f52937fe700 1 -- 172.21.15.116:0/2543153433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528800ed30 con 0x7f5284003a00 2023-11-08T08:19:33.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f529abf7700 1 -- 172.21.15.116:0/2543153433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284003a00 msgr2=0x7f5284003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f529abf7700 1 --2- 172.21.15.116:0/2543153433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284003a00 0x7f5284003e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5288009d30 tx=0x7f5284005880 comp rx=0 tx=0).stop 2023-11-08T08:19:33.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f5299194700 1 -- 172.21.15.116:0/2543153433 reap_dead start 2023-11-08T08:19:33.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f529abf7700 1 -- 172.21.15.116:0/2543153433 shutdown_connections 2023-11-08T08:19:33.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f529abf7700 1 -- 172.21.15.116:0/2543153433 >> 172.21.15.116:0/2543153433 conn(0x7f52940b2fc0 msgr2=0x7f5294071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:33.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f529abf7700 1 -- 172.21.15.116:0/2543153433 shutdown_connections 2023-11-08T08:19:33.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.300+0000 7f529abf7700 1 -- 172.21.15.116:0/2543153433 wait complete. 2023-11-08T08:19:33.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 Processor -- start 2023-11-08T08:19:33.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 -- start start 2023-11-08T08:19:33.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284003a00 0x7f52940cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 0x7f52941f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52941f1ca0 0x7f52940cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940aedb0 con 0x7f52940ce000 2023-11-08T08:19:33.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940af000 con 0x7f52941f1ca0 2023-11-08T08:19:33.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.302+0000 7f529abf7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52940af2e0 con 0x7f5284003a00 2023-11-08T08:19:33.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 0x7f52941f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 0x7f52941f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38056/0 (socket says 172.21.15.116:38056) 2023-11-08T08:19:33.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 -- 172.21.15.116:0/1812760951 learned_addr learned my addr 172.21.15.116:0/1812760951 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:33.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5298993700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284003a00 0x7f52940cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5299194700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52941f1ca0 0x7f52940cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 -- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284003a00 msgr2=0x7f52940cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284003a00 0x7f52940cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 -- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52941f1ca0 msgr2=0x7f52940cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52941f1ca0 0x7f52940cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 -- 172.21.15.116:0/1812760951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940b4eb0 con 0x7f52940ce000 2023-11-08T08:19:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f5293fff700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 0x7f52941f1760 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f528000e780 tx=0x7f528000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280010510 con 0x7f52940ce000 2023-11-08T08:19:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280010690 con 0x7f52940ce000 2023-11-08T08:19:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52940cf790 con 0x7f52940ce000 2023-11-08T08:19:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280010a40 con 0x7f52940ce000 2023-11-08T08:19:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.303+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52940b5040 con 0x7f52940ce000 2023-11-08T08:19:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.304+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52941f20a0 con 0x7f52940ce000 2023-11-08T08:19:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.305+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f528001b030 con 0x7f52940ce000 2023-11-08T08:19:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.305+0000 7f52917fa700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52780682f0 0x7f527806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.305+0000 7f5298993700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52780682f0 0x7f527806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.305+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5280085330 con 0x7f52940ce000 2023-11-08T08:19:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.305+0000 7f5298993700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52780682f0 0x7f527806a7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f52880090b0 tx=0x7f528801f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.309+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5280051b90 con 0x7f52940ce000 2023-11-08T08:19:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.490+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52940743f0 con 0x7f52940ce000 2023-11-08T08:19:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.491+0000 7f52917fa700 1 -- 172.21.15.116:0/1812760951 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f5280019090 con 0x7f52940ce000 2023-11-08T08:19:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:33.493 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:18:43.887182+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:18:40.568047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-08T08:18:41.836252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.493+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52780682f0 msgr2=0x7f527806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.493+0000 7f529abf7700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52780682f0 0x7f527806a7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f52880090b0 tx=0x7f528801f040 comp rx=0 tx=0).stop 2023-11-08T08:19:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.493+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 msgr2=0x7f52941f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.493+0000 7f529abf7700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 0x7f52941f1760 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f528000e780 tx=0x7f528000ea90 comp rx=0 tx=0).stop 2023-11-08T08:19:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 shutdown_connections 2023-11-08T08:19:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52780682f0 0x7f527806a7a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52941f1ca0 0x7f52940cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5284003a00 0x7f52940cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 --2- 172.21.15.116:0/1812760951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52940ce000 0x7f52941f1760 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 >> 172.21.15.116:0/1812760951 conn(0x7f52940b2fc0 msgr2=0x7f529415cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 shutdown_connections 2023-11-08T08:19:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.494+0000 7f529abf7700 1 -- 172.21.15.116:0/1812760951 wait complete. 2023-11-08T08:19:33.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-08T08:19:33.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.628+0000 7f73935c4700 1 Processor -- start 2023-11-08T08:19:33.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.629+0000 7f73935c4700 1 -- start start 2023-11-08T08:19:33.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.629+0000 7f73935c4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f738c0718e0 0x7f738c06c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.629+0000 7f73935c4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f738c06cdc0 0x7f738c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.629+0000 7f73935c4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c160a50 con 0x7f738c15c810 2023-11-08T08:19:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.629+0000 7f73935c4700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738c15cbe0 con 0x7f738c0718e0 2023-11-08T08:19:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.629+0000 7f73935c4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738c160d30 con 0x7f738c06cdc0 2023-11-08T08:19:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7390b5f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f738c0718e0 0x7f738c06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7391360700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f738c15c810 0x7f738c160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48732/0 (socket says 172.21.15.116:48732) 2023-11-08T08:19:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7391360700 1 -- 172.21.15.116:0/2063571820 learned_addr learned my addr 172.21.15.116:0/2063571820 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7383fff700 1 -- 172.21.15.116:0/2063571820 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018868320 0 0) 0x7f738c160a50 con 0x7f738c15c810 2023-11-08T08:19:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7383fff700 1 -- 172.21.15.116:0/2063571820 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73780035b0 con 0x7f738c15c810 2023-11-08T08:19:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7391b61700 1 --2- 172.21.15.116:0/2063571820 >> v2:172.21.15.129:3300/0 conn(0x7f738c06cdc0 0x7f738c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7390b5f700 1 -- 172.21.15.116:0/2063571820 >> v2:172.21.15.129:3300/0 conn(0x7f738c06cdc0 msgr2=0x7f738c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7390b5f700 1 --2- 172.21.15.116:0/2063571820 >> v2:172.21.15.129:3300/0 conn(0x7f738c06cdc0 0x7f738c06a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7390b5f700 1 -- 172.21.15.116:0/2063571820 >> v1:172.21.15.116:6789/0 conn(0x7f738c15c810 legacy=0x7f738c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7390b5f700 1 -- 172.21.15.116:0/2063571820 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c161010 con 0x7f738c0718e0 2023-11-08T08:19:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7390b5f700 1 --2- 172.21.15.116:0/2063571820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0718e0 0x7f738c06c880 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7388009840 tx=0x7f7388009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e3943572a20aa server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7383fff700 1 -- 172.21.15.116:0/2063571820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388011830 con 0x7f738c0718e0 2023-11-08T08:19:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7383fff700 1 -- 172.21.15.116:0/2063571820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7388011990 con 0x7f738c0718e0 2023-11-08T08:19:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.630+0000 7f7383fff700 1 -- 172.21.15.116:0/2063571820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388011d20 con 0x7f738c0718e0 2023-11-08T08:19:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 -- 172.21.15.116:0/2063571820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0718e0 msgr2=0x7f738c06c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 --2- 172.21.15.116:0/2063571820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0718e0 0x7f738c06c880 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7388009840 tx=0x7f7388009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 -- 172.21.15.116:0/2063571820 shutdown_connections 2023-11-08T08:19:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 --2- 172.21.15.116:0/2063571820 >> v2:172.21.15.129:3300/0 conn(0x7f738c06cdc0 0x7f738c06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 --2- 172.21.15.116:0/2063571820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0718e0 0x7f738c06c880 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 -- 172.21.15.116:0/2063571820 >> 172.21.15.116:0/2063571820 conn(0x7f738c06c010 msgr2=0x7f738c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 -- 172.21.15.116:0/2063571820 shutdown_connections 2023-11-08T08:19:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.631+0000 7f73935c4700 1 -- 172.21.15.116:0/2063571820 wait complete. 2023-11-08T08:19:33.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.632+0000 7f73935c4700 1 Processor -- start 2023-11-08T08:19:33.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.632+0000 7f73935c4700 1 -- start start 2023-11-08T08:19:33.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 0x7f738c15f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f738c0718e0 0x7f738c141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f738c15c810 0x7f738c143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c15fda0 con 0x7f738c06cdc0 2023-11-08T08:19:33.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0aed70 con 0x7f738c0718e0 2023-11-08T08:19:33.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0af050 con 0x7f738c15c810 2023-11-08T08:19:33.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 0x7f738c15f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 0x7f738c15f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38080/0 (socket says 172.21.15.116:38080) 2023-11-08T08:19:33.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 -- 172.21.15.116:0/3413664238 learned_addr learned my addr 172.21.15.116:0/3413664238 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:33.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7390b5f700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f738c0718e0 0x7f738c141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391b61700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f738c15c810 0x7f738c143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 -- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f738c15c810 msgr2=0x7f738c143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f738c15c810 0x7f738c143b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 -- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f738c0718e0 msgr2=0x7f738c141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:33.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f738c0718e0 0x7f738c141af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:33.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 -- 172.21.15.116:0/3413664238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880094a0 con 0x7f738c06cdc0 2023-11-08T08:19:33.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7391360700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 0x7f738c15f860 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f737c001eb0 tx=0x7f737c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:33.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c005d70 con 0x7f738c06cdc0 2023-11-08T08:19:33.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c0070d0 con 0x7f738c06cdc0 2023-11-08T08:19:33.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c0ce740 con 0x7f738c06cdc0 2023-11-08T08:19:33.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.633+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c007460 con 0x7f738c06cdc0 2023-11-08T08:19:33.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.634+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c0ceb20 con 0x7f738c06cdc0 2023-11-08T08:19:33.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.634+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c0ce8d0 con 0x7f738c06cdc0 2023-11-08T08:19:33.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.635+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f737c019030 con 0x7f738c06cdc0 2023-11-08T08:19:33.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.635+0000 7f7381ffb700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f73700682f0 0x7f737006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:33.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.635+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f737c085090 con 0x7f738c06cdc0 2023-11-08T08:19:33.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.635+0000 7f7390b5f700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f73700682f0 0x7f737006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:33.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.636+0000 7f7390b5f700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f73700682f0 0x7f737006a7a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f73880090b0 tx=0x7f738801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:33.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.639+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f737c0518f0 con 0x7f738c06cdc0 2023-11-08T08:19:33.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:33.862+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f738c0743f0 con 0x7f738c06cdc0 2023-11-08T08:19:34.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-08T08:19:34.547+0000 7f50f4469700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-08T08:19:34.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.552+0000 7f7381ffb700 1 -- 172.21.15.116:0/3413664238 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f737c054f10 con 0x7f738c06cdc0 2023-11-08T08:19:34.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f73700682f0 msgr2=0x7f737006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:34.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f73700682f0 0x7f737006a7a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f73880090b0 tx=0x7f738801c040 comp rx=0 tx=0).stop 2023-11-08T08:19:34.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 msgr2=0x7f738c15f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:34.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 0x7f738c15f860 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f737c001eb0 tx=0x7f737c00c860 comp rx=0 tx=0).stop 2023-11-08T08:19:34.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 shutdown_connections 2023-11-08T08:19:34.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f738c15c810 0x7f738c143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c06cdc0 0x7f738c15f860 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f73700682f0 0x7f737006a7a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 --2- 172.21.15.116:0/3413664238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f738c0718e0 0x7f738c141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 >> 172.21.15.116:0/3413664238 conn(0x7f738c06c010 msgr2=0x7f738c0c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:34.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 shutdown_connections 2023-11-08T08:19:34.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.555+0000 7f73935c4700 1 -- 172.21.15.116:0/3413664238 wait complete. 2023-11-08T08:19:34.559 INFO:tasks.ceph.mds.d.smithi129.stderr: -2884> 2023-11-08T08:18:48.603+0000 7f3713850700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:19:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-08T08:19:34.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-08T08:19:34.586 INFO:tasks.ceph.mds.d.smithi129.stdout:starting mds.d at 2023-11-08T08:19:34.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.697+0000 7f3527361700 1 Processor -- start 2023-11-08T08:19:34.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.698+0000 7f3527361700 1 -- start start 2023-11-08T08:19:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.698+0000 7f3527361700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3520160300 0x7f352015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.698+0000 7f3527361700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f352015d880 0x7f35200cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.698+0000 7f3527361700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352015ceb0 con 0x7f35200cec00 2023-11-08T08:19:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.698+0000 7f3527361700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35201608c0 con 0x7f3520160300 2023-11-08T08:19:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.698+0000 7f3527361700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3520160ba0 con 0x7f352015d880 2023-11-08T08:19:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35250fd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f35200cec00 0x7f3520065fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48748/0 (socket says 172.21.15.116:48748) 2023-11-08T08:19:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35250fd700 1 -- 172.21.15.116:0/2580008354 learned_addr learned my addr 172.21.15.116:0/2580008354 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35258fe700 1 --2- 172.21.15.116:0/2580008354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f352015d880 0x7f35200cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35248fc700 1 --2- 172.21.15.116:0/2580008354 >> v2:172.21.15.129:3300/0 conn(0x7f3520160300 0x7f352015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f3517fff700 1 -- 172.21.15.116:0/2580008354 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870620065 0 0) 0x7f352015ceb0 con 0x7f35200cec00 2023-11-08T08:19:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f3517fff700 1 -- 172.21.15.116:0/2580008354 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350c0035b0 con 0x7f35200cec00 2023-11-08T08:19:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f3517fff700 1 -- 172.21.15.116:0/2580008354 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3164696233 0 0) 0x7f350c0035b0 con 0x7f35200cec00 2023-11-08T08:19:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35258fe700 1 -- 172.21.15.116:0/2580008354 >> v2:172.21.15.129:3300/0 conn(0x7f3520160300 msgr2=0x7f352015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35258fe700 1 --2- 172.21.15.116:0/2580008354 >> v2:172.21.15.129:3300/0 conn(0x7f3520160300 0x7f352015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35258fe700 1 -- 172.21.15.116:0/2580008354 >> v1:172.21.15.116:6789/0 conn(0x7f35200cec00 legacy=0x7f3520065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35258fe700 1 -- 172.21.15.116:0/2580008354 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3520160e80 con 0x7f352015d880 2023-11-08T08:19:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f35258fe700 1 --2- 172.21.15.116:0/2580008354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f352015d880 0x7f35200cac90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3518009d90 tx=0x7f351800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dff6ae33f886538 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f3517fff700 1 -- 172.21.15.116:0/2580008354 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351800e070 con 0x7f352015d880 2023-11-08T08:19:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f3517fff700 1 -- 172.21.15.116:0/2580008354 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3518003a20 con 0x7f352015d880 2023-11-08T08:19:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.699+0000 7f3517fff700 1 -- 172.21.15.116:0/2580008354 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510001670 con 0x7f352015d880 2023-11-08T08:19:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 -- 172.21.15.116:0/2580008354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f352015d880 msgr2=0x7f35200cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 --2- 172.21.15.116:0/2580008354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f352015d880 0x7f35200cac90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3518009d90 tx=0x7f351800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 -- 172.21.15.116:0/2580008354 shutdown_connections 2023-11-08T08:19:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 --2- 172.21.15.116:0/2580008354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f352015d880 0x7f35200cac90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 --2- 172.21.15.116:0/2580008354 >> v2:172.21.15.129:3300/0 conn(0x7f3520160300 0x7f352015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 -- 172.21.15.116:0/2580008354 >> 172.21.15.116:0/2580008354 conn(0x7f35200cf400 msgr2=0x7f35200cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 -- 172.21.15.116:0/2580008354 shutdown_connections 2023-11-08T08:19:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.700+0000 7f3527361700 1 -- 172.21.15.116:0/2580008354 wait complete. 2023-11-08T08:19:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.701+0000 7f3527361700 1 Processor -- start 2023-11-08T08:19:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.701+0000 7f3527361700 1 -- start start 2023-11-08T08:19:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f3527361700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 0x7f35200b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f3527361700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352015d880 0x7f352014a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f3527361700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3520160300 0x7f35200ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f3527361700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b50f0 con 0x7f35200cec00 2023-11-08T08:19:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f3527361700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200ae550 con 0x7f352015d880 2023-11-08T08:19:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f3527361700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b4ac0 con 0x7f3520160300 2023-11-08T08:19:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 0x7f35200b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 0x7f35200b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38094/0 (socket says 172.21.15.116:38094) 2023-11-08T08:19:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 -- 172.21.15.116:0/2841617757 learned_addr learned my addr 172.21.15.116:0/2841617757 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35248fc700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352015d880 0x7f352014a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35258fe700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3520160300 0x7f35200ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:34.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 -- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3520160300 msgr2=0x7f35200ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:34.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3520160300 0x7f35200ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 -- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352015d880 msgr2=0x7f352014a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:34.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352015d880 0x7f352014a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.702+0000 7f35250fd700 1 -- 172.21.15.116:0/2841617757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3518009a70 con 0x7f35200cec00 2023-11-08T08:19:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.703+0000 7f35250fd700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 0x7f35200b3190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f351000c520 tx=0x7f351000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.703+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510004c80 con 0x7f35200cec00 2023-11-08T08:19:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.703+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3510004de0 con 0x7f35200cec00 2023-11-08T08:19:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.703+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35200cefd0 con 0x7f35200cec00 2023-11-08T08:19:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.703+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510007310 con 0x7f35200cec00 2023-11-08T08:19:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.703+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3520161160 con 0x7f35200cec00 2023-11-08T08:19:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.704+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35200cf160 con 0x7f35200cec00 2023-11-08T08:19:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.704+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f351001a020 con 0x7f35200cec00 2023-11-08T08:19:34.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.704+0000 7f3515ffb700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35040682f0 0x7f350406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:34.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.705+0000 7f35248fc700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35040682f0 0x7f350406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:34.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.705+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3510085820 con 0x7f35200cec00 2023-11-08T08:19:34.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.705+0000 7f35248fc700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35040682f0 0x7f350406a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f35201f5bf0 tx=0x7f351c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:34.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.709+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3510052020 con 0x7f35200cec00 2023-11-08T08:19:34.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:34.933+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f35200ae7e0 con 0x7f35200cec00 2023-11-08T08:19:35.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.554+0000 7f3515ffb700 1 -- 172.21.15.116:0/2841617757 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3510055640 con 0x7f35200cec00 2023-11-08T08:19:35.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35040682f0 msgr2=0x7f350406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:35.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35040682f0 0x7f350406a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f35201f5bf0 tx=0x7f351c009210 comp rx=0 tx=0).stop 2023-11-08T08:19:35.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 msgr2=0x7f35200b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:35.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 0x7f35200b3190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f351000c520 tx=0x7f351000c830 comp rx=0 tx=0).stop 2023-11-08T08:19:35.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 shutdown_connections 2023-11-08T08:19:35.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3520160300 0x7f35200ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35200cec00 0x7f35200b3190 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35040682f0 0x7f350406a7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 --2- 172.21.15.116:0/2841617757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352015d880 0x7f352014a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 >> 172.21.15.116:0/2841617757 conn(0x7f35200cf400 msgr2=0x7f35200cc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:35.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 shutdown_connections 2023-11-08T08:19:35.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.557+0000 7f3527361700 1 -- 172.21.15.116:0/2841617757 wait complete. 2023-11-08T08:19:35.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-08T08:19:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.695+0000 7f4dfeed4700 1 Processor -- start 2023-11-08T08:19:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.695+0000 7f4dfeed4700 1 -- start start 2023-11-08T08:19:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfeed4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4df806a4e0 0x7f4df80cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfeed4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4df80cecf0 0x7f4df80c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfeed4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80ca180 con 0x7f4df806a8b0 2023-11-08T08:19:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfeed4700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df80cf190 con 0x7f4df806a4e0 2023-11-08T08:19:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfeed4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df80cdc50 con 0x7f4df80cecf0 2023-11-08T08:19:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfcc70700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4df806a4e0 0x7f4df80cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfcc70700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4df806a4e0 0x7f4df80cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38110/0 (socket says 172.21.15.116:38110) 2023-11-08T08:19:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfcc70700 1 -- 172.21.15.116:0/687263530 learned_addr learned my addr 172.21.15.116:0/687263530 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4dfcc70700 1 -- 172.21.15.116:0/687263530 >> v2:172.21.15.129:3300/0 conn(0x7f4df80cecf0 msgr2=0x7f4df80c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.696+0000 7f4df7fff700 1 --2- 172.21.15.116:0/687263530 >> v2:172.21.15.129:3300/0 conn(0x7f4df80cecf0 0x7f4df80c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfcc70700 1 --2- 172.21.15.116:0/687263530 >> v2:172.21.15.129:3300/0 conn(0x7f4df80cecf0 0x7f4df80c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfcc70700 1 -- 172.21.15.116:0/687263530 >> v1:172.21.15.129:6789/0 conn(0x7f4df806a8b0 legacy=0x7f4df8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfcc70700 1 -- 172.21.15.116:0/687263530 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80cdf30 con 0x7f4df806a4e0 2023-11-08T08:19:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfcc70700 1 --2- 172.21.15.116:0/687263530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df806a4e0 0x7f4df80cd5c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4df806fd30 tx=0x7f4dec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cc1e67a890c0a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4df77fe700 1 -- 172.21.15.116:0/687263530 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec004230 con 0x7f4df806a4e0 2023-11-08T08:19:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4df77fe700 1 -- 172.21.15.116:0/687263530 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec004390 con 0x7f4df806a4e0 2023-11-08T08:19:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4df77fe700 1 -- 172.21.15.116:0/687263530 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0036a0 con 0x7f4df806a4e0 2023-11-08T08:19:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 -- 172.21.15.116:0/687263530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df806a4e0 msgr2=0x7f4df80cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/687263530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df806a4e0 0x7f4df80cd5c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4df806fd30 tx=0x7f4dec009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 -- 172.21.15.116:0/687263530 shutdown_connections 2023-11-08T08:19:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/687263530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df806a4e0 0x7f4df80cd5c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/687263530 >> v2:172.21.15.129:3300/0 conn(0x7f4df80cecf0 0x7f4df80c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 -- 172.21.15.116:0/687263530 >> 172.21.15.116:0/687263530 conn(0x7f4df80cf370 msgr2=0x7f4df80cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 -- 172.21.15.116:0/687263530 shutdown_connections 2023-11-08T08:19:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.697+0000 7f4dfeed4700 1 -- 172.21.15.116:0/687263530 wait complete. 2023-11-08T08:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 Processor -- start 2023-11-08T08:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 -- start start 2023-11-08T08:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df806a4e0 0x7f4df80c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4df806a8b0 0x7f4df8152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 0x7f4df815b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8159630 con 0x7f4df80cecf0 2023-11-08T08:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8153680 con 0x7f4df806a4e0 2023-11-08T08:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.699+0000 7f4dfeed4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8153960 con 0x7f4df806a8b0 2023-11-08T08:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 0x7f4df815b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 0x7f4df815b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38126/0 (socket says 172.21.15.116:38126) 2023-11-08T08:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 -- 172.21.15.116:0/1041344783 learned_addr learned my addr 172.21.15.116:0/1041344783 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4df7fff700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4df806a8b0 0x7f4df8152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfcc70700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df806a4e0 0x7f4df80c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:35.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 -- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4df806a8b0 msgr2=0x7f4df8152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:35.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4df806a8b0 0x7f4df8152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 -- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df806a4e0 msgr2=0x7f4df80c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:35.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df806a4e0 0x7f4df80c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:35.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 -- 172.21.15.116:0/1041344783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec009550 con 0x7f4df80cecf0 2023-11-08T08:19:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfd471700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 0x7f4df815b520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4df000cc10 tx=0x7f4df000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df000aa60 con 0x7f4df80cecf0 2023-11-08T08:19:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df000abc0 con 0x7f4df80cecf0 2023-11-08T08:19:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df80ce4f0 con 0x7f4df80cecf0 2023-11-08T08:19:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0012420 con 0x7f4df80cecf0 2023-11-08T08:19:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.700+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df80ce210 con 0x7f4df80cecf0 2023-11-08T08:19:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.701+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df80ce7d0 con 0x7f4df80cecf0 2023-11-08T08:19:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.702+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4df0010050 con 0x7f4df80cecf0 2023-11-08T08:19:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.702+0000 7f4df57fa700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ddc0682f0 0x7f4ddc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.702+0000 7f4dfcc70700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ddc0682f0 0x7f4ddc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.702+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4df0085ea0 con 0x7f4df80cecf0 2023-11-08T08:19:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.702+0000 7f4dfcc70700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ddc0682f0 0x7f4ddc06a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009130 tx=0x7f4dec008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.706+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4df0052640 con 0x7f4df80cecf0 2023-11-08T08:19:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:35.887+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4df80743f0 con 0x7f4df80cecf0 2023-11-08T08:19:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.564+0000 7f4df57fa700 1 -- 172.21.15.116:0/1041344783 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v22) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4df0055c60 con 0x7f4df80cecf0 2023-11-08T08:19:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ddc0682f0 msgr2=0x7f4ddc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ddc0682f0 0x7f4ddc06a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009130 tx=0x7f4dec008030 comp rx=0 tx=0).stop 2023-11-08T08:19:36.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 msgr2=0x7f4df815b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 0x7f4df815b520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4df000cc10 tx=0x7f4df000cf20 comp rx=0 tx=0).stop 2023-11-08T08:19:36.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 shutdown_connections 2023-11-08T08:19:36.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df80cecf0 0x7f4df815b520 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4ddc0682f0 0x7f4ddc06a7a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df806a4e0 0x7f4df80c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 --2- 172.21.15.116:0/1041344783 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4df806a8b0 0x7f4df8152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.567+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 >> 172.21.15.116:0/1041344783 conn(0x7f4df80cf370 msgr2=0x7f4df80c8450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.568+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 shutdown_connections 2023-11-08T08:19:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.569+0000 7f4dfeed4700 1 -- 172.21.15.116:0/1041344783 wait complete. 2023-11-08T08:19:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2023-11-08T08:19:36.587 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-08T08:19:36.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.714+0000 7f7d7e866700 1 Processor -- start 2023-11-08T08:19:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.715+0000 7f7d7e866700 1 -- start start 2023-11-08T08:19:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.715+0000 7f7d7e866700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d78161390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.715+0000 7f7d7e866700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d78157730 con 0x7f7d78071320 2023-11-08T08:19:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.715+0000 7f7d7e866700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d78160f00 con 0x7f7d7815dca0 2023-11-08T08:19:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.715+0000 7f7d7e866700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780cdba0 con 0x7f7d78157360 2023-11-08T08:19:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d7ce03700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7d78071320 0x7f7d780ac8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48758/0 (socket says 172.21.15.116:48758) 2023-11-08T08:19:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d7ce03700 1 -- 172.21.15.116:0/401815166 learned_addr learned my addr 172.21.15.116:0/401815166 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d77fff700 1 --2- 172.21.15.116:0/401815166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d78161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d76ffd700 1 -- 172.21.15.116:0/401815166 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836660640 0 0) 0x7f7d78157730 con 0x7f7d78071320 2023-11-08T08:19:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d76ffd700 1 -- 172.21.15.116:0/401815166 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d600035b0 con 0x7f7d78071320 2023-11-08T08:19:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d77fff700 1 -- 172.21.15.116:0/401815166 >> v1:172.21.15.129:6789/0 conn(0x7f7d7815dca0 legacy=0x7f7d78070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d77fff700 1 -- 172.21.15.116:0/401815166 >> v1:172.21.15.116:6789/0 conn(0x7f7d78071320 legacy=0x7f7d780ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d76ffd700 1 -- 172.21.15.116:0/401815166 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543523394 0 0) 0x7f7d78160f00 con 0x7f7d7815dca0 2023-11-08T08:19:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d77fff700 1 -- 172.21.15.116:0/401815166 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d781608b0 con 0x7f7d78157360 2023-11-08T08:19:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d77fff700 1 --2- 172.21.15.116:0/401815166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d78161390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009870 tx=0x7f7d6c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3fbbade5781511 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d76ffd700 1 -- 172.21.15.116:0/401815166 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c011830 con 0x7f7d78157360 2023-11-08T08:19:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d76ffd700 1 -- 172.21.15.116:0/401815166 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c011990 con 0x7f7d78157360 2023-11-08T08:19:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.716+0000 7f7d76ffd700 1 -- 172.21.15.116:0/401815166 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c011d00 con 0x7f7d78157360 2023-11-08T08:19:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 -- 172.21.15.116:0/401815166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 msgr2=0x7f7d78161390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 --2- 172.21.15.116:0/401815166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d78161390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009870 tx=0x7f7d6c009b80 comp rx=0 tx=0).stop 2023-11-08T08:19:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 -- 172.21.15.116:0/401815166 shutdown_connections 2023-11-08T08:19:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 --2- 172.21.15.116:0/401815166 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d78161390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 -- 172.21.15.116:0/401815166 >> 172.21.15.116:0/401815166 conn(0x7f7d780cfc30 msgr2=0x7f7d780d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 -- 172.21.15.116:0/401815166 shutdown_connections 2023-11-08T08:19:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.717+0000 7f7d7e866700 1 -- 172.21.15.116:0/401815166 wait complete. 2023-11-08T08:19:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.718+0000 7f7d7e866700 1 Processor -- start 2023-11-08T08:19:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.718+0000 7f7d7e866700 1 -- start start 2023-11-08T08:19:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7e866700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78071320 0x7f7d7814b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7e866700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d780ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7e866700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 0x7f7d780ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7e866700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7814beb0 con 0x7f7d7815dca0 2023-11-08T08:19:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7e866700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780b4db0 con 0x7f7d78071320 2023-11-08T08:19:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7e866700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780b5090 con 0x7f7d78157360 2023-11-08T08:19:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 0x7f7d780ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 0x7f7d780ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38138/0 (socket says 172.21.15.116:38138) 2023-11-08T08:19:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 -- 172.21.15.116:0/2507901973 learned_addr learned my addr 172.21.15.116:0/2507901973 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d777fe700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d780ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d77fff700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78071320 0x7f7d7814b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 -- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 msgr2=0x7f7d780ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d780ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 -- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78071320 msgr2=0x7f7d7814b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78071320 0x7f7d7814b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d7ce03700 1 -- 172.21.15.116:0/2507901973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0094a0 con 0x7f7d7815dca0 2023-11-08T08:19:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.719+0000 7f7d777fe700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d780ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.720+0000 7f7d7ce03700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 0x7f7d780ade70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000ca70 tx=0x7f7d7000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.720+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000aaa0 con 0x7f7d7815dca0 2023-11-08T08:19:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.720+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7000ac00 con 0x7f7d7815dca0 2023-11-08T08:19:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.720+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d780af410 con 0x7f7d7815dca0 2023-11-08T08:19:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.720+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70012400 con 0x7f7d7815dca0 2023-11-08T08:19:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.720+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d78160b90 con 0x7f7d7815dca0 2023-11-08T08:19:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.721+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7815e110 con 0x7f7d7815dca0 2023-11-08T08:19:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.721+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d70019030 con 0x7f7d7815dca0 2023-11-08T08:19:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.722+0000 7f7d74ff9700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7d640682f0 0x7f7d6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.722+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5710+0+0 (secure 0 0 0) 0x7f7d70085620 con 0x7f7d7815dca0 2023-11-08T08:19:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.722+0000 7f7d77fff700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7d640682f0 0x7f7d6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.722+0000 7f7d77fff700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7d640682f0 0x7f7d6406a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0097c0 tx=0x7f7d6c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.726+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d7004b5e0 con 0x7f7d7815dca0 2023-11-08T08:19:36.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:36.906+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7d780743f0 con 0x7f7d7815dca0 2023-11-08T08:19:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.566+0000 7f7d74ff9700 1 -- 172.21.15.116:0/2507901973 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7d70052030 con 0x7f7d7815dca0 2023-11-08T08:19:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7d640682f0 msgr2=0x7f7d6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7d640682f0 0x7f7d6406a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0097c0 tx=0x7f7d6c008030 comp rx=0 tx=0).stop 2023-11-08T08:19:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 msgr2=0x7f7d780ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 0x7f7d780ade70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000ca70 tx=0x7f7d7000ce30 comp rx=0 tx=0).stop 2023-11-08T08:19:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 shutdown_connections 2023-11-08T08:19:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7815dca0 0x7f7d780ade70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7d640682f0 0x7f7d6406a7a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78071320 0x7f7d7814b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 --2- 172.21.15.116:0/2507901973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7d78157360 0x7f7d780ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.571+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 >> 172.21.15.116:0/2507901973 conn(0x7f7d780cfc30 msgr2=0x7f7d780b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.572+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 shutdown_connections 2023-11-08T08:19:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.572+0000 7f7d7e866700 1 -- 172.21.15.116:0/2507901973 wait complete. 2023-11-08T08:19:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2023-11-08T08:19:37.589 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-08T08:19:37.590 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-08T08:19:37.590 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-08T08:19:37.590 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:37.591 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-08T08:19:37.591 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-08T08:19:37.591 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-08T08:19:37.592 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:37.592 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-08T08:19:37.592 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-08T08:19:37.593 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-08T08:19:37.593 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:37.593 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-08T08:19:37.593 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-08T08:19:37.594 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-08T08:19:37.594 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:19:37.600 INFO:tasks.ceph.mds.a:Stopped 2023-11-08T08:19:37.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-08T08:19:37.602 INFO:tasks.ceph.mds.c:Stopped 2023-11-08T08:19:37.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-08T08:19:37.606 INFO:tasks.ceph.mds.b:Stopped 2023-11-08T08:19:37.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-08T08:19:37.608 INFO:tasks.ceph.mds.d:Stopped 2023-11-08T08:19:37.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-08T08:19:37.645 INFO:tasks.ceph.mds.a:Started 2023-11-08T08:19:37.646 DEBUG:teuthology.parallel:result is None 2023-11-08T08:19:37.646 INFO:tasks.ceph.mds.c:Started 2023-11-08T08:19:37.647 DEBUG:teuthology.parallel:result is None 2023-11-08T08:19:37.649 INFO:tasks.ceph.mds.b:Started 2023-11-08T08:19:37.649 DEBUG:teuthology.parallel:result is None 2023-11-08T08:19:37.650 INFO:tasks.ceph.mds.d:Started 2023-11-08T08:19:37.651 DEBUG:teuthology.parallel:result is None 2023-11-08T08:19:37.651 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-08T08:19:37.737 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-08T08:19:37.737 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-08T08:19:37.742 INFO:tasks.ceph.mds.d.smithi129.stdout:starting mds.d at 2023-11-08T08:19:37.742 INFO:tasks.ceph.mds.b.smithi129.stdout:starting mds.b at 2023-11-08T08:19:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.794+0000 7f2b768ee700 1 Processor -- start 2023-11-08T08:19:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.794+0000 7f2b768ee700 1 -- start start 2023-11-08T08:19:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b768ee700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2b7015c8c0 0x7f2b70160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b768ee700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b7006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b768ee700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700d2d30 con 0x7f2b7006b050 2023-11-08T08:19:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b768ee700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b7006b4d0 con 0x7f2b7015c8c0 2023-11-08T08:19:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b768ee700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700d0b40 con 0x7f2b700b5dc0 2023-11-08T08:19:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b7006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6ffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2b7015c8c0 0x7f2b70160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b74e8b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f2b7006b050 0x7f2b700d0fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:39328/0 (socket says 172.21.15.116:39328) 2023-11-08T08:19:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b74e8b700 1 -- 172.21.15.116:0/1957708894 learned_addr learned my addr 172.21.15.116:0/1957708894 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:37.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6effd700 1 -- 172.21.15.116:0/1957708894 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079135607 0 0) 0x7f2b68001490 con 0x7f2b7006b050 2023-11-08T08:19:37.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6effd700 1 -- 172.21.15.116:0/1957708894 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c0035b0 con 0x7f2b7006b050 2023-11-08T08:19:37.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 msgr2=0x7f2b7006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:37.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6ffff700 1 --2- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b7006ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1957708894 >> v1:172.21.15.129:6789/0 conn(0x7f2b7006b050 legacy=0x7f2b700d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1957708894 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b701611f0 con 0x7f2b7015c8c0 2023-11-08T08:19:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.795+0000 7f2b6f7fe700 1 --2- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b7006ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b6ffff700 1 --2- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b70160cb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009840 tx=0x7f2b58009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d06618667c5640 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b6effd700 1 -- 172.21.15.116:0/1957708894 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58011830 con 0x7f2b7015c8c0 2023-11-08T08:19:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b6effd700 1 -- 172.21.15.116:0/1957708894 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b58011990 con 0x7f2b7015c8c0 2023-11-08T08:19:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b6effd700 1 -- 172.21.15.116:0/1957708894 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58011d20 con 0x7f2b7015c8c0 2023-11-08T08:19:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 -- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 msgr2=0x7f2b70160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b70160cb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009840 tx=0x7f2b58009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 -- 172.21.15.116:0/1957708894 shutdown_connections 2023-11-08T08:19:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b70160cb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1957708894 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b7006ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 -- 172.21.15.116:0/1957708894 >> 172.21.15.116:0/1957708894 conn(0x7f2b7006c520 msgr2=0x7f2b7006c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:37.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 -- 172.21.15.116:0/1957708894 shutdown_connections 2023-11-08T08:19:37.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.796+0000 7f2b768ee700 1 -- 172.21.15.116:0/1957708894 wait complete. 2023-11-08T08:19:37.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.797+0000 7f2b768ee700 1 Processor -- start 2023-11-08T08:19:37.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 -- start start 2023-11-08T08:19:37.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 0x7f2b701558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:37.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b70155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:37.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b700ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:37.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7015c190 con 0x7f2b7006b050 2023-11-08T08:19:37.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7015e0e0 con 0x7f2b7015c8c0 2023-11-08T08:19:37.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b768ee700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700ca0e0 con 0x7f2b700b5dc0 2023-11-08T08:19:37.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b6ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 0x7f2b701558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:37.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b6ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 0x7f2b701558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38166/0 (socket says 172.21.15.116:38166) 2023-11-08T08:19:37.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.798+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1969263561 learned_addr learned my addr 172.21.15.116:0/1969263561 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:37.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b74e8b700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b700ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:37.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6f7fe700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b70155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 msgr2=0x7f2b70155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6ffff700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b70155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 msgr2=0x7f2b700ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6ffff700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b700ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6ffff700 1 -- 172.21.15.116:0/1969263561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b600094a0 con 0x7f2b7006b050 2023-11-08T08:19:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6f7fe700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b70155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b74e8b700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b700ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6ffff700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 0x7f2b701558f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2b580090b0 tx=0x7f2b58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580045d0 con 0x7f2b7006b050 2023-11-08T08:19:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5801e070 con 0x7f2b7006b050 2023-11-08T08:19:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b580094a0 con 0x7f2b7006b050 2023-11-08T08:19:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580038e0 con 0x7f2b7006b050 2023-11-08T08:19:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.799+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b700cd760 con 0x7f2b7006b050 2023-11-08T08:19:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.800+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b700d2a50 con 0x7f2b7006b050 2023-11-08T08:19:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.800+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b5800f050 con 0x7f2b7006b050 2023-11-08T08:19:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.801+0000 7f2b6cff9700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2b500682c0 0x7f2b5006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.801+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f2b58086920 con 0x7f2b7006b050 2023-11-08T08:19:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.801+0000 7f2b6f7fe700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2b500682c0 0x7f2b5006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.801+0000 7f2b6f7fe700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2b500682c0 0x7f2b5006a770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b70156c50 tx=0x7f2b60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:37.805+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b58053480 con 0x7f2b7006b050 2023-11-08T08:19:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.000+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2b70067d70 con 0x7f2b7006b050 2023-11-08T08:19:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.783+0000 7f2b6cff9700 1 -- 172.21.15.116:0/1969263561 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2b58015080 con 0x7f2b7006b050 2023-11-08T08:19:38.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.785+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2b500682c0 msgr2=0x7f2b5006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.785+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2b500682c0 0x7f2b5006a770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b70156c50 tx=0x7f2b60008040 comp rx=0 tx=0).stop 2023-11-08T08:19:38.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 msgr2=0x7f2b701558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 0x7f2b701558f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2b580090b0 tx=0x7f2b58004750 comp rx=0 tx=0).stop 2023-11-08T08:19:38.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 shutdown_connections 2023-11-08T08:19:38.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2b500682c0 0x7f2b5006a770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b7015c8c0 0x7f2b700ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b7006b050 0x7f2b701558f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 --2- 172.21.15.116:0/1969263561 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2b700b5dc0 0x7f2b70155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 >> 172.21.15.116:0/1969263561 conn(0x7f2b7006c520 msgr2=0x7f2b700d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:38.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 shutdown_connections 2023-11-08T08:19:38.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.786+0000 7f2b768ee700 1 -- 172.21.15.116:0/1969263561 wait complete. 2023-11-08T08:19:38.790 INFO:teuthology.orchestra.run.smithi116.stderr: removed all blocklist entries 2023-11-08T08:19:38.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-08T08:19:38.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.923+0000 7f13c4166700 1 Processor -- start 2023-11-08T08:19:38.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.923+0000 7f13c4166700 1 -- start start 2023-11-08T08:19:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c4166700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f13bc0b4b00 0x7f13bc0cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c4166700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b4fc0 con 0x7f13bc06feb0 2023-11-08T08:19:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c4166700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b05e0 con 0x7f13bc0ced70 2023-11-08T08:19:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c4166700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc159c30 con 0x7f13bc0b4b00 2023-11-08T08:19:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c1701700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13bc06feb0 0x7f13bc0cd1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48786/0 (socket says 172.21.15.116:48786) 2023-11-08T08:19:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c1701700 1 -- 172.21.15.116:0/3532194415 learned_addr learned my addr 172.21.15.116:0/3532194415 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123020208 0 0) 0x7f13bc0b4fc0 con 0x7f13bc06feb0 2023-11-08T08:19:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a80035b0 con 0x7f13bc06feb0 2023-11-08T08:19:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.924+0000 7f13c2703700 1 --2- 172.21.15.116:0/3532194415 >> v2:172.21.15.129:3300/0 conn(0x7f13bc0b4b00 0x7f13bc0cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714275568 0 0) 0x7f13a80035b0 con 0x7f13bc06feb0 2023-11-08T08:19:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 >> v2:172.21.15.129:3300/0 conn(0x7f13bc0b4b00 msgr2=0x7f13bc0cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 --2- 172.21.15.116:0/3532194415 >> v2:172.21.15.129:3300/0 conn(0x7f13bc0b4b00 0x7f13bc0cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 >> v1:172.21.15.129:6789/0 conn(0x7f13bc0ced70 legacy=0x7f13bc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0b08c0 con 0x7f13bc06feb0 2023-11-08T08:19:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f13b8001670 con 0x7f13bc06feb0 2023-11-08T08:19:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 >> v1:172.21.15.116:6789/0 conn(0x7f13bc06feb0 legacy=0x7f13bc0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a8003a00 0x7f13a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a8004850 0x7f13a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc159c30 con 0x7f13a8005200 2023-11-08T08:19:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a800c080 con 0x7f13a8004850 2023-11-08T08:19:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a800c240 con 0x7f13a8003a00 2023-11-08T08:19:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c1701700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a8004850 0x7f13a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c1f02700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a8003a00 0x7f13a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 -- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a8003a00 msgr2=0x7f13a8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13a8003a00 0x7f13a8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 -- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a8004850 msgr2=0x7f13a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13a8004850 0x7f13a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 -- 172.21.15.116:0/3532194415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80035b0 con 0x7f13a8005200 2023-11-08T08:19:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 -- 172.21.15.116:0/3532194415 reap_dead start 2023-11-08T08:19:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c2703700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13a800ba10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009fc0 tx=0x7f13a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3807bf7b293a48f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4003a20 con 0x7f13a8005200 2023-11-08T08:19:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400f040 con 0x7f13a8005200 2023-11-08T08:19:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.925+0000 7f13c0f00700 1 -- 172.21.15.116:0/3532194415 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4003de0 con 0x7f13a8005200 2023-11-08T08:19:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 -- 172.21.15.116:0/3532194415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 msgr2=0x7f13a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13a800ba10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009fc0 tx=0x7f13a8005880 comp rx=0 tx=0).stop 2023-11-08T08:19:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 -- 172.21.15.116:0/3532194415 shutdown_connections 2023-11-08T08:19:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 --2- 172.21.15.116:0/3532194415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13a800ba10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 -- 172.21.15.116:0/3532194415 >> 172.21.15.116:0/3532194415 conn(0x7f13bc0b2fc0 msgr2=0x7f13bc071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 -- 172.21.15.116:0/3532194415 shutdown_connections 2023-11-08T08:19:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.926+0000 7f13c4166700 1 -- 172.21.15.116:0/3532194415 wait complete. 2023-11-08T08:19:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.927+0000 7f13c4166700 1 Processor -- start 2023-11-08T08:19:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.927+0000 7f13c4166700 1 -- start start 2023-11-08T08:19:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c4166700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13bc153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c4166700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13bc1f5060 0x7f13bc1f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c4166700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13bc155be0 0x7f13bc156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c4166700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0ce430 con 0x7f13a8005200 2023-11-08T08:19:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c4166700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1f5a10 con 0x7f13bc1f5060 2023-11-08T08:19:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c4166700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc156880 con 0x7f13bc155be0 2023-11-08T08:19:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13bc153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13bc153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38184/0 (socket says 172.21.15.116:38184) 2023-11-08T08:19:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 -- 172.21.15.116:0/2006411238 learned_addr learned my addr 172.21.15.116:0/2006411238 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 -- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13bc155be0 msgr2=0x7f13bc156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c2703700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13bc155be0 0x7f13bc156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1701700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13bc1f5060 0x7f13bc1f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13bc155be0 0x7f13bc156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 -- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13bc1f5060 msgr2=0x7f13bc1f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13bc1f5060 0x7f13bc1f54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1f02700 1 -- 172.21.15.116:0/2006411238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0b5260 con 0x7f13a8005200 2023-11-08T08:19:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c2703700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13bc155be0 0x7f13bc156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.928+0000 7f13c1701700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13bc1f5060 0x7f13bc1f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.929+0000 7f13c1f02700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13bc153df0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00e500 tx=0x7f13ac00e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.929+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0053a0 con 0x7f13a8005200 2023-11-08T08:19:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.929+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac005500 con 0x7f13a8005200 2023-11-08T08:19:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.929+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc0b0ba0 con 0x7f13a8005200 2023-11-08T08:19:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.929+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0038a0 con 0x7f13a8005200 2023-11-08T08:19:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.929+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc072420 con 0x7f13a8005200 2023-11-08T08:19:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.930+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc0cce70 con 0x7f13a8005200 2023-11-08T08:19:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.930+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13ac01b030 con 0x7f13a8005200 2023-11-08T08:19:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.930+0000 7f13b27fc700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a00682f0 0x7f13a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.930+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f13ac084f10 con 0x7f13a8005200 2023-11-08T08:19:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.930+0000 7f13c1701700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a00682f0 0x7f13a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:38.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.931+0000 7f13c1701700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a00682f0 0x7f13a006a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0ac600 tx=0x7f13b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:38.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:38.935+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13ac04d5e0 con 0x7f13a8005200 2023-11-08T08:19:39.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.185+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f13bc067370 con 0x7f13a8005200 2023-11-08T08:19:39.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.185+0000 7f13b27fc700 1 -- 172.21.15.116:0/2006411238 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f13ac019090 con 0x7f13a8005200 2023-11-08T08:19:39.188 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:39.188 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:19:39.188 INFO:teuthology.orchestra.run.smithi116.stdout: "auth_dump": [ 2023-11-08T08:19:39.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.188 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.a", 2023-11-08T08:19:39.189 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlwPmvARAAdJKHnQqKVPWnN7uQiHc+WQ==", 2023-11-08T08:19:39.189 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.189 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.189 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:19:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:19:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.b", 2023-11-08T08:19:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlXUbqDRAATJnFxEiS//echgEFyiS+iw==", 2023-11-08T08:19:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:19:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:19:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.c", 2023-11-08T08:19:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlNA7+CBAAkUCXKhY1qacysaUprxlU9w==", 2023-11-08T08:19:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:19:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:19:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.195 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.196 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.d", 2023-11-08T08:19:39.196 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtltlgoFRAAwAkFfezamgYHcC9awLpLpA==", 2023-11-08T08:19:39.196 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.196 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.197 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:19:39.197 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:19:39.197 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.198 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.199 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.0", 2023-11-08T08:19:39.199 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAyREtlwCT6MBAA0hUY6XxZE8Q/cW66704LAA==", 2023-11-08T08:19:39.200 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.200 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.200 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.201 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.202 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.1", 2023-11-08T08:19:39.202 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA1REtlHxOBLxAAHR12A9jsXL7W8TeG55VeWg==", 2023-11-08T08:19:39.203 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.203 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.204 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.204 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.205 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.2", 2023-11-08T08:19:39.205 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA4REtlcPidLhAAhNEoyldnbVRtP9IQr08wag==", 2023-11-08T08:19:39.206 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.206 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.206 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.207 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.207 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.3", 2023-11-08T08:19:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA7REtlH/3XLBAASJ532Or0yrTC8qfr3kCziQ==", 2023-11-08T08:19:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.4", 2023-11-08T08:19:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBBREtl5+beLxAAXM67eNV2/FzHU4+gp3zMng==", 2023-11-08T08:19:39.210 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.210 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.210 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.211 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.211 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.5", 2023-11-08T08:19:39.212 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBEREtl4pChLhAA470kPHr4WFBGgwEi0CB95A==", 2023-11-08T08:19:39.212 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.212 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.212 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.212 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.213 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.6", 2023-11-08T08:19:39.213 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBHREtlh5g/LRAAY7eRgyAdWK0ExY6Qaf6hgQ==", 2023-11-08T08:19:39.213 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.214 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.214 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.214 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.215 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.7", 2023-11-08T08:19:39.215 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBKREtly78MKxAA5+d+0MekiMWQK1sAn1Ue6w==", 2023-11-08T08:19:39.216 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.216 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:19:39.216 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:19:39.217 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.217 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.0", 2023-11-08T08:19:39.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlHhh+GRAAKHMDRBv0YayhqERZiWXGAw==", 2023-11-08T08:19:39.218 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.218 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.218 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:19:39.219 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:19:39.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:19:39.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.220 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.1", 2023-11-08T08:19:39.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtl2ICCIhAAdgVi5BOXs1xqGLKgTLEXoQ==", 2023-11-08T08:19:39.221 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.222 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.222 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:19:39.222 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:19:39.222 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:19:39.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.223 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.2", 2023-11-08T08:19:39.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlKh8kHxAA/Ear8UA/wixquw0KBzet2A==", 2023-11-08T08:19:39.223 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.224 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.224 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:19:39.224 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:19:39.224 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:19:39.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.226 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.3", 2023-11-08T08:19:39.226 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtl17vQJRAA/R0BvGRHgw8ZySwL4KyuGg==", 2023-11-08T08:19:39.226 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.227 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:19:39.227 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:19:39.228 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:19:39.228 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:19:39.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.229 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.229 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.admin", 2023-11-08T08:19:39.230 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtlwc+XHhAA8ruR6iJNLTqLSXROZ9iRHQ==", 2023-11-08T08:19:39.230 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.230 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:19:39.231 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:19:39.231 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:19:39.231 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.233 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-mds", 2023-11-08T08:19:39.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBQREtl2T75KBAAVPEEaXMTTqwxQEljgRxCwQ==", 2023-11-08T08:19:39.233 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.233 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-mds" 2023-11-08T08:19:39.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.234 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-mgr", 2023-11-08T08:19:39.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBQREtlcUf5KBAA4dacgAveuVzVeFTiii8Bcw==", 2023-11-08T08:19:39.235 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.235 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-08T08:19:39.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.236 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-osd", 2023-11-08T08:19:39.236 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBQREtl3FD5KBAAMEH5IhfSr4/yeziSpJKbCQ==", 2023-11-08T08:19:39.236 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.236 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-osd" 2023-11-08T08:19:39.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.237 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-rbd", 2023-11-08T08:19:39.237 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBQREtlnVn5KBAAq++grktmKcAr8bzpN/Jfag==", 2023-11-08T08:19:39.237 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.237 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-08T08:19:39.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.238 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-08T08:19:39.239 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBQREtlA2H5KBAAvrnUwdrz7BFh0dWOEa+xxg==", 2023-11-08T08:19:39.239 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.239 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-08T08:19:39.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.240 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-rgw", 2023-11-08T08:19:39.240 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBQREtll2j5KBAARQLWG0BpuYnTIkEXzrSGWg==", 2023-11-08T08:19:39.240 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.240 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-08T08:19:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.x", 2023-11-08T08:19:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtl1TH6ORAAQMEd7GQJFqk28L6HfaYPlg==", 2023-11-08T08:19:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:19:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-08T08:19:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.y", 2023-11-08T08:19:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtlXRIMNBAAC9TkKhc8QzTE+DK2eDs/bg==", 2023-11-08T08:19:39.244 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:19:39.244 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:19:39.244 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-08T08:19:39.244 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:19:39.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:39.245 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-11-08T08:19:39.245 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:19:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a00682f0 msgr2=0x7f13a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a00682f0 0x7f13a006a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0ac600 tx=0x7f13b8009040 comp rx=0 tx=0).stop 2023-11-08T08:19:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 msgr2=0x7f13bc153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13bc153df0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00e500 tx=0x7f13ac00e8c0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 shutdown_connections 2023-11-08T08:19:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f13bc155be0 0x7f13bc156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a8005200 0x7f13bc153df0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f13a00682f0 0x7f13a006a7a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 --2- 172.21.15.116:0/2006411238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13bc1f5060 0x7f13bc1f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 >> 172.21.15.116:0/2006411238 conn(0x7f13bc0b2fc0 msgr2=0x7f13bc06ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 shutdown_connections 2023-11-08T08:19:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.187+0000 7f13c4166700 1 -- 172.21.15.116:0/2006411238 wait complete. 2023-11-08T08:19:39.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-08T08:19:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.369+0000 7f8b98981700 1 Processor -- start 2023-11-08T08:19:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.369+0000 7f8b98981700 1 -- start start 2023-11-08T08:19:39.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.370+0000 7f8b98981700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b940af0a0 0x7f8b9415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.370+0000 7f8b98981700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b94159ec0 con 0x7f8b940b1650 2023-11-08T08:19:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.370+0000 7f8b98981700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9415d8a0 con 0x7f8b941599b0 2023-11-08T08:19:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.370+0000 7f8b98981700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9415db80 con 0x7f8b940af0a0 2023-11-08T08:19:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b92d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b940af0a0 0x7f8b9415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9259c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8b940b1650 0x7f8b940b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48800/0 (socket says 172.21.15.116:48800) 2023-11-08T08:19:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9259c700 1 -- 172.21.15.116:0/3597766947 learned_addr learned my addr 172.21.15.116:0/3597766947 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9159a700 1 -- 172.21.15.116:0/3597766947 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810809966 0 0) 0x7f8b94159ec0 con 0x7f8b940b1650 2023-11-08T08:19:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9159a700 1 -- 172.21.15.116:0/3597766947 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c0035b0 con 0x7f8b940b1650 2023-11-08T08:19:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b92d9d700 1 -- 172.21.15.116:0/3597766947 >> v1:172.21.15.129:6789/0 conn(0x7f8b941599b0 legacy=0x7f8b9415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b92d9d700 1 -- 172.21.15.116:0/3597766947 >> v1:172.21.15.116:6789/0 conn(0x7f8b940b1650 legacy=0x7f8b940b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b92d9d700 1 -- 172.21.15.116:0/3597766947 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9415de60 con 0x7f8b940af0a0 2023-11-08T08:19:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b92d9d700 1 --2- 172.21.15.116:0/3597766947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b940af0a0 0x7f8b9415a350 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009d90 tx=0x7f8b8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f50e6d473b5227 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9159a700 1 -- 172.21.15.116:0/3597766947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00e070 con 0x7f8b940af0a0 2023-11-08T08:19:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9159a700 1 -- 172.21.15.116:0/3597766947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c00bb20 con 0x7f8b940af0a0 2023-11-08T08:19:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.371+0000 7f8b9159a700 1 -- 172.21.15.116:0/3597766947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c012430 con 0x7f8b940af0a0 2023-11-08T08:19:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 -- 172.21.15.116:0/3597766947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b940af0a0 msgr2=0x7f8b9415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 --2- 172.21.15.116:0/3597766947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b940af0a0 0x7f8b9415a350 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009d90 tx=0x7f8b8c00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 -- 172.21.15.116:0/3597766947 shutdown_connections 2023-11-08T08:19:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 --2- 172.21.15.116:0/3597766947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b940af0a0 0x7f8b9415a350 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 -- 172.21.15.116:0/3597766947 >> 172.21.15.116:0/3597766947 conn(0x7f8b9406c010 msgr2=0x7f8b940cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 -- 172.21.15.116:0/3597766947 shutdown_connections 2023-11-08T08:19:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.372+0000 7f8b98981700 1 -- 172.21.15.116:0/3597766947 wait complete. 2023-11-08T08:19:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.373+0000 7f8b98981700 1 Processor -- start 2023-11-08T08:19:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.373+0000 7f8b98981700 1 -- start start 2023-11-08T08:19:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b98981700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b940af0a0 0x7f8b941544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b98981700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b940b1650 0x7f8b94140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b98981700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 0x7f8b940b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b98981700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9413e730 con 0x7f8b94140680 2023-11-08T08:19:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b98981700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9413e0b0 con 0x7f8b940af0a0 2023-11-08T08:19:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b98981700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94158160 con 0x7f8b940b1650 2023-11-08T08:19:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 0x7f8b940b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 0x7f8b940b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38210/0 (socket says 172.21.15.116:38210) 2023-11-08T08:19:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 -- 172.21.15.116:0/270010543 learned_addr learned my addr 172.21.15.116:0/270010543 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 -- 172.21.15.116:0/270010543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b940b1650 msgr2=0x7f8b94140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b940b1650 0x7f8b94140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 -- 172.21.15.116:0/270010543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b940af0a0 msgr2=0x7f8b941544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b940af0a0 0x7f8b941544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 -- 172.21.15.116:0/270010543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c009a70 con 0x7f8b94140680 2023-11-08T08:19:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b92d9d700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 0x7f8b940b4a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b940708b0 tx=0x7f8b8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c012430 con 0x7f8b94140680 2023-11-08T08:19:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.374+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c020070 con 0x7f8b94140680 2023-11-08T08:19:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.375+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9415aa20 con 0x7f8b94140680 2023-11-08T08:19:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.375+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c004280 con 0x7f8b94140680 2023-11-08T08:19:39.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.375+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9415a890 con 0x7f8b94140680 2023-11-08T08:19:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.375+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b940d0a40 con 0x7f8b94140680 2023-11-08T08:19:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.376+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b8c0043e0 con 0x7f8b94140680 2023-11-08T08:19:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.376+0000 7f8b82ffd700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b740682f0 0x7f8b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.376+0000 7f8b9259c700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b740682f0 0x7f8b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.376+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8b8c022070 con 0x7f8b94140680 2023-11-08T08:19:39.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.377+0000 7f8b9259c700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b740682f0 0x7f8b7406a7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8b880016b0 tx=0x7f8b880079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:39.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.381+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b8c0548a0 con 0x7f8b94140680 2023-11-08T08:19:39.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.640+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f8b9413e870 con 0x7f8b94140680 2023-11-08T08:19:39.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.643+0000 7f8b82ffd700 1 -- 172.21.15.116:0/270010543 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8b9413e870 con 0x7f8b94140680 2023-11-08T08:19:39.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b740682f0 msgr2=0x7f8b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b740682f0 0x7f8b7406a7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8b880016b0 tx=0x7f8b880079c0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 msgr2=0x7f8b940b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 0x7f8b940b4a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b940708b0 tx=0x7f8b8c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 shutdown_connections 2023-11-08T08:19:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b94140680 0x7f8b940b4a10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b740682f0 0x7f8b7406a7a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b940af0a0 0x7f8b941544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 --2- 172.21.15.116:0/270010543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b940b1650 0x7f8b94140140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 >> 172.21.15.116:0/270010543 conn(0x7f8b9406c010 msgr2=0x7f8b9406a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:39.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 shutdown_connections 2023-11-08T08:19:39.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.645+0000 7f8b98981700 1 -- 172.21.15.116:0/270010543 wait complete. 2023-11-08T08:19:39.658 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-08T08:19:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.778+0000 7fcaab3bf700 1 Processor -- start 2023-11-08T08:19:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.778+0000 7fcaab3bf700 1 -- start start 2023-11-08T08:19:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaab3bf700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcaa415c810 0x7fcaa4160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaab3bf700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcaa406a270 0x7fcaa406a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaab3bf700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa406abb0 con 0x7fcaa40718e0 2023-11-08T08:19:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaab3bf700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa415cbe0 con 0x7fcaa406a270 2023-11-08T08:19:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaab3bf700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa41607a0 con 0x7fcaa415c810 2023-11-08T08:19:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaa995c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcaa406a270 0x7fcaa406a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaa895a700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcaa40718e0 0x7fcaa406c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48802/0 (socket says 172.21.15.116:48802) 2023-11-08T08:19:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaa995c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcaa406a270 0x7fcaa406a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38220/0 (socket says 172.21.15.116:38220) 2023-11-08T08:19:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.779+0000 7fcaa895a700 1 -- 172.21.15.116:0/880414682 learned_addr learned my addr 172.21.15.116:0/880414682 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaa995c700 1 -- 172.21.15.116:0/880414682 >> v2:172.21.15.129:3300/0 conn(0x7fcaa415c810 msgr2=0x7fcaa4160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaa915b700 1 --2- 172.21.15.116:0/880414682 >> v2:172.21.15.129:3300/0 conn(0x7fcaa415c810 0x7fcaa4160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaa995c700 1 --2- 172.21.15.116:0/880414682 >> v2:172.21.15.129:3300/0 conn(0x7fcaa415c810 0x7fcaa4160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaa995c700 1 -- 172.21.15.116:0/880414682 >> v1:172.21.15.116:6789/0 conn(0x7fcaa40718e0 legacy=0x7fcaa406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaa995c700 1 -- 172.21.15.116:0/880414682 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4160a50 con 0x7fcaa406a270 2023-11-08T08:19:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fca9bfff700 1 -- 172.21.15.116:0/880414682 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380683608 0 0) 0x7fcaa406abb0 con 0x7fcaa40718e0 2023-11-08T08:19:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaa995c700 1 --2- 172.21.15.116:0/880414682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa406a270 0x7fcaa406a670 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40b27c0 tx=0x7fca9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fc0039fcf76a7e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fca9bfff700 1 -- 172.21.15.116:0/880414682 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00e070 con 0x7fcaa406a270 2023-11-08T08:19:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fca9bfff700 1 -- 172.21.15.116:0/880414682 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c00bbd0 con 0x7fcaa406a270 2023-11-08T08:19:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fca9bfff700 1 -- 172.21.15.116:0/880414682 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c012430 con 0x7fcaa406a270 2023-11-08T08:19:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaab3bf700 1 -- 172.21.15.116:0/880414682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa406a270 msgr2=0x7fcaa406a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/880414682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa406a270 0x7fcaa406a670 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40b27c0 tx=0x7fca9c00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaab3bf700 1 -- 172.21.15.116:0/880414682 shutdown_connections 2023-11-08T08:19:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/880414682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa406a270 0x7fcaa406a670 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/880414682 >> v2:172.21.15.129:3300/0 conn(0x7fcaa415c810 0x7fcaa4160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.780+0000 7fcaab3bf700 1 -- 172.21.15.116:0/880414682 >> 172.21.15.116:0/880414682 conn(0x7fcaa406c010 msgr2=0x7fcaa40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.781+0000 7fcaab3bf700 1 -- 172.21.15.116:0/880414682 shutdown_connections 2023-11-08T08:19:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.781+0000 7fcaab3bf700 1 -- 172.21.15.116:0/880414682 wait complete. 2023-11-08T08:19:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 Processor -- start 2023-11-08T08:19:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 -- start start 2023-11-08T08:19:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa406a270 0x7fcaa415bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcaa40718e0 0x7fcaa41f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 0x7fcaa41ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa41f23b0 con 0x7fcaa415c810 2023-11-08T08:19:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa41f2030 con 0x7fcaa406a270 2023-11-08T08:19:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaab3bf700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa41f5b60 con 0x7fcaa40718e0 2023-11-08T08:19:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.782+0000 7fcaa995c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 0x7fcaa41ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 0x7fcaa41ef2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38234/0 (socket says 172.21.15.116:38234) 2023-11-08T08:19:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 -- 172.21.15.116:0/4035923243 learned_addr learned my addr 172.21.15.116:0/4035923243 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:39.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa915b700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa406a270 0x7fcaa415bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa895a700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcaa40718e0 0x7fcaa41f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 -- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcaa40718e0 msgr2=0x7fcaa41f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcaa40718e0 0x7fcaa41f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 -- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa406a270 msgr2=0x7fcaa415bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa406a270 0x7fcaa415bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 -- 172.21.15.116:0/4035923243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c009a70 con 0x7fcaa415c810 2023-11-08T08:19:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa915b700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa406a270 0x7fcaa415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa895a700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcaa40718e0 0x7fcaa41f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaa995c700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 0x7fcaa41ef2e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4067450 tx=0x7fca9c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c012430 con 0x7fcaa415c810 2023-11-08T08:19:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c020070 con 0x7fcaa415c810 2023-11-08T08:19:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c004280 con 0x7fcaa415c810 2023-11-08T08:19:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa4161010 con 0x7fcaa415c810 2023-11-08T08:19:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.783+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa4160d30 con 0x7fcaa415c810 2023-11-08T08:19:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.784+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa40ceb20 con 0x7fcaa415c810 2023-11-08T08:19:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.785+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca9c02e070 con 0x7fcaa415c810 2023-11-08T08:19:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.785+0000 7fca99ffb700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca880682f0 0x7fca8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.785+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fca9c022070 con 0x7fcaa415c810 2023-11-08T08:19:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.785+0000 7fcaa915b700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca880682f0 0x7fca8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.785+0000 7fcaa915b700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca880682f0 0x7fca8806a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fca94006fd0 tx=0x7fca94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:39.789+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca9c054400 con 0x7fcaa415c810 2023-11-08T08:19:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.043+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fcaa41f24f0 con 0x7fcaa415c810 2023-11-08T08:19:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.047+0000 7fca99ffb700 1 -- 172.21.15.116:0/4035923243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fca9c02c0a0 con 0x7fcaa415c810 2023-11-08T08:19:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca880682f0 msgr2=0x7fca8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca880682f0 0x7fca8806a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fca94006fd0 tx=0x7fca94008040 comp rx=0 tx=0).stop 2023-11-08T08:19:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 msgr2=0x7fcaa41ef2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 0x7fcaa41ef2e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4067450 tx=0x7fca9c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 shutdown_connections 2023-11-08T08:19:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa415c810 0x7fcaa41ef2e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fca880682f0 0x7fca8806a7a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa406a270 0x7fcaa415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 --2- 172.21.15.116:0/4035923243 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcaa40718e0 0x7fcaa41f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 >> 172.21.15.116:0/4035923243 conn(0x7fcaa406c010 msgr2=0x7fcaa406db30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 shutdown_connections 2023-11-08T08:19:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.049+0000 7fcaab3bf700 1 -- 172.21.15.116:0/4035923243 wait complete. 2023-11-08T08:19:40.062 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-08T08:19:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.183+0000 7f8a99d0c700 1 Processor -- start 2023-11-08T08:19:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.183+0000 7f8a99d0c700 1 -- start start 2023-11-08T08:19:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a99d0c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8a941599b0 0x7f8a9415d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a99d0c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a94159ec0 con 0x7f8a940b1650 2023-11-08T08:19:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a99d0c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9415aa60 con 0x7f8a940af0a0 2023-11-08T08:19:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a99d0c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9415d8a0 con 0x7f8a941599b0 2023-11-08T08:19:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a937fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8a940b1650 0x7f8a940b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48810/0 (socket says 172.21.15.116:48810) 2023-11-08T08:19:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a937fe700 1 -- 172.21.15.116:0/3734137035 learned_addr learned my addr 172.21.15.116:0/3734137035 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622647241 0 0) 0x7f8a94159ec0 con 0x7f8a940b1650 2023-11-08T08:19:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a800035b0 con 0x7f8a940b1650 2023-11-08T08:19:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.184+0000 7f8a92ffd700 1 --2- 172.21.15.116:0/3734137035 >> v2:172.21.15.129:3300/0 conn(0x7f8a941599b0 0x7f8a9415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107219914 0 0) 0x7f8a800035b0 con 0x7f8a940b1650 2023-11-08T08:19:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 >> v2:172.21.15.129:3300/0 conn(0x7f8a941599b0 msgr2=0x7f8a9415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 --2- 172.21.15.116:0/3734137035 >> v2:172.21.15.129:3300/0 conn(0x7f8a941599b0 0x7f8a9415d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 >> v1:172.21.15.129:6789/0 conn(0x7f8a940af0a0 legacy=0x7f8a9415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9415db80 con 0x7f8a940b1650 2023-11-08T08:19:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f8a7c001670 con 0x7f8a940b1650 2023-11-08T08:19:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a92ffd700 1 --2- 172.21.15.116:0/3734137035 >> v2:172.21.15.129:3300/0 conn(0x7f8a941599b0 0x7f8a9415d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 >> v1:172.21.15.116:6789/0 conn(0x7f8a940b1650 legacy=0x7f8a940b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a80003a00 0x7f8a80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80004850 0x7f8a80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80005200 0x7f8a8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9415d8a0 con 0x7f8a80005200 2023-11-08T08:19:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000c190 con 0x7f8a80003a00 2023-11-08T08:19:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000c350 con 0x7f8a80004850 2023-11-08T08:19:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80005200 0x7f8a8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a937fe700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a80003a00 0x7f8a80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a92ffd700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80004850 0x7f8a80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 -- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80004850 msgr2=0x7f8a80004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80004850 0x7f8a80004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 -- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a80003a00 msgr2=0x7f8a80003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a80003a00 0x7f8a80003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 -- 172.21.15.116:0/3734137035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800035b0 con 0x7f8a80005200 2023-11-08T08:19:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 -- 172.21.15.116:0/3734137035 reap_dead start 2023-11-08T08:19:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a93fff700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80005200 0x7f8a8000ba10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00c160 tx=0x7f8a80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9ef81a4aed1a5f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c001db0 con 0x7f8a80005200 2023-11-08T08:19:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c00cca0 con 0x7f8a80005200 2023-11-08T08:19:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.185+0000 7f8a927fc700 1 -- 172.21.15.116:0/3734137035 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c003c20 con 0x7f8a80005200 2023-11-08T08:19:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 -- 172.21.15.116:0/3734137035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80005200 msgr2=0x7f8a8000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80005200 0x7f8a8000ba10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00c160 tx=0x7f8a80005880 comp rx=0 tx=0).stop 2023-11-08T08:19:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 -- 172.21.15.116:0/3734137035 shutdown_connections 2023-11-08T08:19:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/3734137035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80005200 0x7f8a8000ba10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 -- 172.21.15.116:0/3734137035 >> 172.21.15.116:0/3734137035 conn(0x7f8a9406c010 msgr2=0x7f8a940cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 -- 172.21.15.116:0/3734137035 shutdown_connections 2023-11-08T08:19:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.186+0000 7f8a99d0c700 1 -- 172.21.15.116:0/3734137035 wait complete. 2023-11-08T08:19:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.187+0000 7f8a99d0c700 1 Processor -- start 2023-11-08T08:19:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.187+0000 7f8a99d0c700 1 -- start start 2023-11-08T08:19:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a99d0c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80005200 0x7f8a941e2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a99d0c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 0x7f8a941f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a99d0c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a941e44d0 0x7f8a940b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a99d0c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a941e3430 con 0x7f8a940c5a00 2023-11-08T08:19:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a99d0c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a941e3710 con 0x7f8a941e44d0 2023-11-08T08:19:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a99d0c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a941e39f0 con 0x7f8a80005200 2023-11-08T08:19:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 0x7f8a941f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 0x7f8a941f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38264/0 (socket says 172.21.15.116:38264) 2023-11-08T08:19:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 -- 172.21.15.116:0/873464260 learned_addr learned my addr 172.21.15.116:0/873464260 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 -- 172.21.15.116:0/873464260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80005200 msgr2=0x7f8a941e2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80005200 0x7f8a941e2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 -- 172.21.15.116:0/873464260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a941e44d0 msgr2=0x7f8a940b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a941e44d0 0x7f8a940b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 -- 172.21.15.116:0/873464260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9415a350 con 0x7f8a940c5a00 2023-11-08T08:19:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.188+0000 7f8a92ffd700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 0x7f8a941f33f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8a84010890 tx=0x7f8a84010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.189+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400e8c0 con 0x7f8a940c5a00 2023-11-08T08:19:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.189+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8400ea20 con 0x7f8a940c5a00 2023-11-08T08:19:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.189+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9415de60 con 0x7f8a940c5a00 2023-11-08T08:19:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.189+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400edd0 con 0x7f8a940c5a00 2023-11-08T08:19:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.189+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9415a4e0 con 0x7f8a940c5a00 2023-11-08T08:19:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.189+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a940743f0 con 0x7f8a940c5a00 2023-11-08T08:19:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.190+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a84026070 con 0x7f8a940c5a00 2023-11-08T08:19:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.190+0000 7f8a8bfff700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8a74068320 0x7f8a7406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.190+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8a84086390 con 0x7f8a940c5a00 2023-11-08T08:19:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.190+0000 7f8a937fe700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8a74068320 0x7f8a7406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.191+0000 7f8a937fe700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8a74068320 0x7f8a7406a7d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0011e0 tx=0x7f8a7c001170 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.194+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a84052f40 con 0x7f8a940c5a00 2023-11-08T08:19:40.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.447+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f8a9406e620 con 0x7f8a940c5a00 2023-11-08T08:19:40.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.450+0000 7f8a8bfff700 1 -- 172.21.15.116:0/873464260 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8a84056560 con 0x7f8a940c5a00 2023-11-08T08:19:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8a74068320 msgr2=0x7f8a7406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8a74068320 0x7f8a7406a7d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0011e0 tx=0x7f8a7c001170 comp rx=0 tx=0).stop 2023-11-08T08:19:40.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 msgr2=0x7f8a941f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 0x7f8a941f33f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8a84010890 tx=0x7f8a84010c50 comp rx=0 tx=0).stop 2023-11-08T08:19:40.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 shutdown_connections 2023-11-08T08:19:40.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8a74068320 0x7f8a7406a7d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a941e44d0 0x7f8a940b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8a80005200 0x7f8a941e2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 --2- 172.21.15.116:0/873464260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a940c5a00 0x7f8a941f33f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 >> 172.21.15.116:0/873464260 conn(0x7f8a9406c010 msgr2=0x7f8a94070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:40.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 shutdown_connections 2023-11-08T08:19:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.452+0000 7f8a99d0c700 1 -- 172.21.15.116:0/873464260 wait complete. 2023-11-08T08:19:40.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-08T08:19:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.586+0000 7fe37741d700 1 Processor -- start 2023-11-08T08:19:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.586+0000 7fe37741d700 1 -- start start 2023-11-08T08:19:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe37741d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe370160300 0x7fe37015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe37741d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe37015d880 0x7fe3700cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe37741d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37015ceb0 con 0x7fe3700cec00 2023-11-08T08:19:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe37741d700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3701608c0 con 0x7fe37015d880 2023-11-08T08:19:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe37741d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe370160ba0 con 0x7fe370160300 2023-11-08T08:19:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe37015d880 0x7fe3700cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe37015d880 0x7fe3700cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38270/0 (socket says 172.21.15.116:38270) 2023-11-08T08:19:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 -- 172.21.15.116:0/3446136838 learned_addr learned my addr 172.21.15.116:0/3446136838 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3749b8700 1 --2- 172.21.15.116:0/3446136838 >> v2:172.21.15.129:3300/0 conn(0x7fe370160300 0x7fe37015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 -- 172.21.15.116:0/3446136838 >> v2:172.21.15.129:3300/0 conn(0x7fe370160300 msgr2=0x7fe37015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 --2- 172.21.15.116:0/3446136838 >> v2:172.21.15.129:3300/0 conn(0x7fe370160300 0x7fe37015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 -- 172.21.15.116:0/3446136838 >> v1:172.21.15.129:6789/0 conn(0x7fe3700cec00 legacy=0x7fe370065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 -- 172.21.15.116:0/3446136838 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe370160e80 con 0x7fe37015d880 2023-11-08T08:19:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe3759ba700 1 --2- 172.21.15.116:0/3446136838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe37015d880 0x7fe3700cac90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe37006fd20 tx=0x7fe36800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30151f26b5492ff server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe367fff700 1 -- 172.21.15.116:0/3446136838 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36800e070 con 0x7fe37015d880 2023-11-08T08:19:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe367fff700 1 -- 172.21.15.116:0/3446136838 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36800b9c0 con 0x7fe37015d880 2023-11-08T08:19:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.587+0000 7fe367fff700 1 -- 172.21.15.116:0/3446136838 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36800bd70 con 0x7fe37015d880 2023-11-08T08:19:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 -- 172.21.15.116:0/3446136838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe37015d880 msgr2=0x7fe3700cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 --2- 172.21.15.116:0/3446136838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe37015d880 0x7fe3700cac90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe37006fd20 tx=0x7fe36800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 -- 172.21.15.116:0/3446136838 shutdown_connections 2023-11-08T08:19:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 --2- 172.21.15.116:0/3446136838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe37015d880 0x7fe3700cac90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 --2- 172.21.15.116:0/3446136838 >> v2:172.21.15.129:3300/0 conn(0x7fe370160300 0x7fe37015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 -- 172.21.15.116:0/3446136838 >> 172.21.15.116:0/3446136838 conn(0x7fe3700cf400 msgr2=0x7fe3700cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 -- 172.21.15.116:0/3446136838 shutdown_connections 2023-11-08T08:19:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.588+0000 7fe37741d700 1 -- 172.21.15.116:0/3446136838 wait complete. 2023-11-08T08:19:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.589+0000 7fe37741d700 1 Processor -- start 2023-11-08T08:19:40.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.589+0000 7fe37741d700 1 -- start start 2023-11-08T08:19:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe37741d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3700cec00 0x7fe3700c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe37741d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe37015d880 0x7fe370155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe37741d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 0x7fe3701f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe37741d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3701f3180 con 0x7fe370160300 2023-11-08T08:19:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe37741d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe370159830 con 0x7fe3700cec00 2023-11-08T08:19:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe37741d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe370156060 con 0x7fe37015d880 2023-11-08T08:19:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 0x7fe3701f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 0x7fe3701f55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38286/0 (socket says 172.21.15.116:38286) 2023-11-08T08:19:40.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 -- 172.21.15.116:0/1156091513 learned_addr learned my addr 172.21.15.116:0/1156091513 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:40.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3751b9700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3700cec00 0x7fe3700c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3749b8700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe37015d880 0x7fe370155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 -- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe37015d880 msgr2=0x7fe370155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe37015d880 0x7fe370155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 -- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3700cec00 msgr2=0x7fe3700c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3700cec00 0x7fe3700c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 -- 172.21.15.116:0/1156091513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368009b30 con 0x7fe370160300 2023-11-08T08:19:40.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3751b9700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3700cec00 0x7fe3700c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:40.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe3759ba700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 0x7fe3701f55b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe36800b370 tx=0x7fe368004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.590+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3680099b0 con 0x7fe370160300 2023-11-08T08:19:40.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.591+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3680092e0 con 0x7fe370160300 2023-11-08T08:19:40.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.591+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36801c660 con 0x7fe370160300 2023-11-08T08:19:40.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.591+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3700cefd0 con 0x7fe370160300 2023-11-08T08:19:40.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.591+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe370161160 con 0x7fe370160300 2023-11-08T08:19:40.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.592+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3700cf160 con 0x7fe370160300 2023-11-08T08:19:40.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.592+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe36802c070 con 0x7fe370160300 2023-11-08T08:19:40.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.593+0000 7fe365ffb700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe3540682f0 0x7fe35406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.593+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe3680874f0 con 0x7fe370160300 2023-11-08T08:19:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.593+0000 7fe3751b9700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe3540682f0 0x7fe35406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.593+0000 7fe3751b9700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe3540682f0 0x7fe35406a7a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe36c000f60 tx=0x7fe36c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.597+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3680540a0 con 0x7fe370160300 2023-11-08T08:19:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.849+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe3700743f0 con 0x7fe370160300 2023-11-08T08:19:40.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.853+0000 7fe365ffb700 1 -- 172.21.15.116:0/1156091513 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe3680576c0 con 0x7fe370160300 2023-11-08T08:19:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe3540682f0 msgr2=0x7fe35406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe3540682f0 0x7fe35406a7a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe36c000f60 tx=0x7fe36c009040 comp rx=0 tx=0).stop 2023-11-08T08:19:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 msgr2=0x7fe3701f55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 0x7fe3701f55b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe36800b370 tx=0x7fe368004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 shutdown_connections 2023-11-08T08:19:40.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe370160300 0x7fe3701f55b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe3540682f0 0x7fe35406a7a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3700cec00 0x7fe3700c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 --2- 172.21.15.116:0/1156091513 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe37015d880 0x7fe370155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 >> 172.21.15.116:0/1156091513 conn(0x7fe3700cf400 msgr2=0x7fe37006afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:40.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 shutdown_connections 2023-11-08T08:19:40.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.855+0000 7fe37741d700 1 -- 172.21.15.116:0/1156091513 wait complete. 2023-11-08T08:19:40.868 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-08T08:19:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.988+0000 7fab76ae7700 1 Processor -- start 2023-11-08T08:19:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.989+0000 7fab76ae7700 1 -- start start 2023-11-08T08:19:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.989+0000 7fab76ae7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fab700cec00 0x7fab70065fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.989+0000 7fab76ae7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fab70160300 0x7fab7015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.989+0000 7fab76ae7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7015ceb0 con 0x7fab7015d880 2023-11-08T08:19:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.989+0000 7fab76ae7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab701608c0 con 0x7fab70160300 2023-11-08T08:19:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.989+0000 7fab76ae7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab70160ba0 con 0x7fab700cec00 2023-11-08T08:19:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fab70160300 0x7fab7015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab75084700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fab7015d880 0x7fab700cac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48818/0 (socket says 172.21.15.116:48818) 2023-11-08T08:19:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab75084700 1 -- 172.21.15.116:0/2968350863 learned_addr learned my addr 172.21.15.116:0/2968350863 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab74883700 1 --2- 172.21.15.116:0/2968350863 >> v2:172.21.15.129:3300/0 conn(0x7fab700cec00 0x7fab70065fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6ffff700 1 -- 172.21.15.116:0/2968350863 >> v2:172.21.15.129:3300/0 conn(0x7fab700cec00 msgr2=0x7fab70065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6ffff700 1 --2- 172.21.15.116:0/2968350863 >> v2:172.21.15.129:3300/0 conn(0x7fab700cec00 0x7fab70065fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6ffff700 1 -- 172.21.15.116:0/2968350863 >> v1:172.21.15.116:6789/0 conn(0x7fab7015d880 legacy=0x7fab700cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6ffff700 1 -- 172.21.15.116:0/2968350863 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70160e80 con 0x7fab70160300 2023-11-08T08:19:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6f7fe700 1 -- 172.21.15.116:0/2968350863 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449895344 0 0) 0x7fab7015ceb0 con 0x7fab7015d880 2023-11-08T08:19:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6ffff700 1 --2- 172.21.15.116:0/2968350863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70160300 0x7fab7015d340 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fab700cceb0 tx=0x7fab5c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc914b5c5b93f372 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6f7fe700 1 -- 172.21.15.116:0/2968350863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c011830 con 0x7fab70160300 2023-11-08T08:19:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6f7fe700 1 -- 172.21.15.116:0/2968350863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c011990 con 0x7fab70160300 2023-11-08T08:19:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.990+0000 7fab6f7fe700 1 -- 172.21.15.116:0/2968350863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c011d20 con 0x7fab70160300 2023-11-08T08:19:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 -- 172.21.15.116:0/2968350863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70160300 msgr2=0x7fab7015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 --2- 172.21.15.116:0/2968350863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70160300 0x7fab7015d340 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fab700cceb0 tx=0x7fab5c009b20 comp rx=0 tx=0).stop 2023-11-08T08:19:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 -- 172.21.15.116:0/2968350863 shutdown_connections 2023-11-08T08:19:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 --2- 172.21.15.116:0/2968350863 >> v2:172.21.15.129:3300/0 conn(0x7fab700cec00 0x7fab70065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 --2- 172.21.15.116:0/2968350863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70160300 0x7fab7015d340 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 -- 172.21.15.116:0/2968350863 >> 172.21.15.116:0/2968350863 conn(0x7fab700cf400 msgr2=0x7fab700cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 -- 172.21.15.116:0/2968350863 shutdown_connections 2023-11-08T08:19:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.991+0000 7fab76ae7700 1 -- 172.21.15.116:0/2968350863 wait complete. 2023-11-08T08:19:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.992+0000 7fab76ae7700 1 Processor -- start 2023-11-08T08:19:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.992+0000 7fab76ae7700 1 -- start start 2023-11-08T08:19:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab76ae7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fab700cec00 0x7fab701f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab76ae7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 0x7fab7015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab76ae7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab70160300 0x7fab700b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab76ae7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c5d10 con 0x7fab7015d880 2023-11-08T08:19:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab76ae7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c5370 con 0x7fab70160300 2023-11-08T08:19:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab76ae7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c5650 con 0x7fab700cec00 2023-11-08T08:19:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab6ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 0x7fab7015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab6ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 0x7fab7015e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38296/0 (socket says 172.21.15.116:38296) 2023-11-08T08:19:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.993+0000 7fab6ffff700 1 -- 172.21.15.116:0/3535929386 learned_addr learned my addr 172.21.15.116:0/3535929386 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab75084700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab70160300 0x7fab700b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab74883700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fab700cec00 0x7fab701f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6ffff700 1 -- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fab700cec00 msgr2=0x7fab701f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6ffff700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fab700cec00 0x7fab701f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6ffff700 1 -- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab70160300 msgr2=0x7fab700b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6ffff700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab70160300 0x7fab700b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6ffff700 1 -- 172.21.15.116:0/3535929386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5c0094a0 con 0x7fab7015d880 2023-11-08T08:19:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab75084700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab70160300 0x7fab700b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab74883700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fab700cec00 0x7fab701f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6ffff700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 0x7fab7015e3e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009b60 tx=0x7fab5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0038a0 con 0x7fab7015d880 2023-11-08T08:19:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c01e070 con 0x7fab7015d880 2023-11-08T08:19:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab70066510 con 0x7fab7015d880 2023-11-08T08:19:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c003c00 con 0x7fab7015d880 2023-11-08T08:19:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.994+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab70161160 con 0x7fab7015d880 2023-11-08T08:19:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.995+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700cefd0 con 0x7fab7015d880 2023-11-08T08:19:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.995+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab5c00f050 con 0x7fab7015d880 2023-11-08T08:19:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.996+0000 7fab6d7fa700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fab540682f0 0x7fab5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.996+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fab5c0869d0 con 0x7fab7015d880 2023-11-08T08:19:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.996+0000 7fab74883700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fab540682f0 0x7fab5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:40.996+0000 7fab74883700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fab540682f0 0x7fab5406a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fab64009a00 tx=0x7fab64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.000+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab5c053580 con 0x7fab7015d880 2023-11-08T08:19:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.253+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fab700743f0 con 0x7fab7015d880 2023-11-08T08:19:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.256+0000 7fab6d7fa700 1 -- 172.21.15.116:0/3535929386 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fab5c0150a0 con 0x7fab7015d880 2023-11-08T08:19:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fab540682f0 msgr2=0x7fab5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fab540682f0 0x7fab5406a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fab64009a00 tx=0x7fab64008040 comp rx=0 tx=0).stop 2023-11-08T08:19:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 msgr2=0x7fab7015e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 0x7fab7015e3e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009b60 tx=0x7fab5c004750 comp rx=0 tx=0).stop 2023-11-08T08:19:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 shutdown_connections 2023-11-08T08:19:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fab540682f0 0x7fab5406a7a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab70160300 0x7fab700b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fab700cec00 0x7fab701f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 --2- 172.21.15.116:0/3535929386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab7015d880 0x7fab7015e3e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 >> 172.21.15.116:0/3535929386 conn(0x7fab700cf400 msgr2=0x7fab7006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 shutdown_connections 2023-11-08T08:19:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.258+0000 7fab76ae7700 1 -- 172.21.15.116:0/3535929386 wait complete. 2023-11-08T08:19:41.272 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-08T08:19:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.391+0000 7fe6ab9d0700 1 Processor -- start 2023-11-08T08:19:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6ab9d0700 1 -- start start 2023-11-08T08:19:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6ab9d0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6a415c810 0x7fe6a4160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6ab9d0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe6a40718e0 0x7fe6a406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6ab9d0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a41607a0 con 0x7fe6a406cdc0 2023-11-08T08:19:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6ab9d0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a415cbe0 con 0x7fe6a415c810 2023-11-08T08:19:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6ab9d0700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a4160a50 con 0x7fe6a40718e0 2023-11-08T08:19:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6a976c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6a415c810 0x7fe6a4160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6a976c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6a415c810 0x7fe6a4160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38300/0 (socket says 172.21.15.116:38300) 2023-11-08T08:19:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.392+0000 7fe6a976c700 1 -- 172.21.15.116:0/1113976732 learned_addr learned my addr 172.21.15.116:0/1113976732 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6a976c700 1 -- 172.21.15.116:0/1113976732 >> v2:172.21.15.129:3300/0 conn(0x7fe6a40718e0 msgr2=0x7fe6a406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6a8f6b700 1 --2- 172.21.15.116:0/1113976732 >> v2:172.21.15.129:3300/0 conn(0x7fe6a40718e0 0x7fe6a406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6a976c700 1 --2- 172.21.15.116:0/1113976732 >> v2:172.21.15.129:3300/0 conn(0x7fe6a40718e0 0x7fe6a406c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6a976c700 1 -- 172.21.15.116:0/1113976732 >> v1:172.21.15.129:6789/0 conn(0x7fe6a406cdc0 legacy=0x7fe6a406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6a976c700 1 -- 172.21.15.116:0/1113976732 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a4160d30 con 0x7fe6a415c810 2023-11-08T08:19:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6a976c700 1 --2- 172.21.15.116:0/1113976732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a415c810 0x7fe6a4160260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40cc380 tx=0x7fe694009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c84317c838afcc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe69bfff700 1 -- 172.21.15.116:0/1113976732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694004230 con 0x7fe6a415c810 2023-11-08T08:19:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe69bfff700 1 -- 172.21.15.116:0/1113976732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe694004390 con 0x7fe6a415c810 2023-11-08T08:19:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe69bfff700 1 -- 172.21.15.116:0/1113976732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6940036a0 con 0x7fe6a415c810 2023-11-08T08:19:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/1113976732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a415c810 msgr2=0x7fe6a4160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.393+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/1113976732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a415c810 0x7fe6a4160260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40cc380 tx=0x7fe694009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.394+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/1113976732 shutdown_connections 2023-11-08T08:19:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.394+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/1113976732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a415c810 0x7fe6a4160260 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.394+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/1113976732 >> v2:172.21.15.129:3300/0 conn(0x7fe6a40718e0 0x7fe6a406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.394+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/1113976732 >> 172.21.15.116:0/1113976732 conn(0x7fe6a406c010 msgr2=0x7fe6a40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.394+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/1113976732 shutdown_connections 2023-11-08T08:19:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.394+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/1113976732 wait complete. 2023-11-08T08:19:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 Processor -- start 2023-11-08T08:19:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 -- start start 2023-11-08T08:19:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6a406cdc0 0x7fe6a41f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 0x7fe6a40aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6a415c810 0x7fe6a41f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40b4490 con 0x7fe6a40718e0 2023-11-08T08:19:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40b1bf0 con 0x7fe6a415c810 2023-11-08T08:19:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.395+0000 7fe6ab9d0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a40b3af0 con 0x7fe6a406cdc0 2023-11-08T08:19:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 0x7fe6a40aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 0x7fe6a40aef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38306/0 (socket says 172.21.15.116:38306) 2023-11-08T08:19:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 -- 172.21.15.116:0/4175947235 learned_addr learned my addr 172.21.15.116:0/4175947235 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a9f6d700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6a415c810 0x7fe6a41f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a976c700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6a406cdc0 0x7fe6a41f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 -- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6a406cdc0 msgr2=0x7fe6a41f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6a406cdc0 0x7fe6a41f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 -- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6a415c810 msgr2=0x7fe6a41f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6a415c810 0x7fe6a41f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 -- 172.21.15.116:0/4175947235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694009550 con 0x7fe6a40718e0 2023-11-08T08:19:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a976c700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6a406cdc0 0x7fe6a41f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6a8f6b700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 0x7fe6a40aef40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000c7f0 tx=0x7fe6a000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0011800 con 0x7fe6a40718e0 2023-11-08T08:19:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0011960 con 0x7fe6a40718e0 2023-11-08T08:19:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a40ceb20 con 0x7fe6a40718e0 2023-11-08T08:19:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0011d10 con 0x7fe6a40718e0 2023-11-08T08:19:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.396+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a4161010 con 0x7fe6a40718e0 2023-11-08T08:19:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.397+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a40ce740 con 0x7fe6a40718e0 2023-11-08T08:19:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.397+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe6a0021070 con 0x7fe6a40718e0 2023-11-08T08:19:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.398+0000 7fe699ffb700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6880682f0 0x7fe68806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.398+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe6a0085840 con 0x7fe6a40718e0 2023-11-08T08:19:41.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.398+0000 7fe6a976c700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6880682f0 0x7fe68806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.398+0000 7fe6a976c700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6880682f0 0x7fe68806a7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4152880 tx=0x7fe694008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.402+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6a00523f0 con 0x7fe6a40718e0 2023-11-08T08:19:41.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.658+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fe6a40743f0 con 0x7fe6a40718e0 2023-11-08T08:19:41.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.661+0000 7fe699ffb700 1 -- 172.21.15.116:0/4175947235 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe6a0055a10 con 0x7fe6a40718e0 2023-11-08T08:19:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6880682f0 msgr2=0x7fe68806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6880682f0 0x7fe68806a7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4152880 tx=0x7fe694008030 comp rx=0 tx=0).stop 2023-11-08T08:19:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 msgr2=0x7fe6a40aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 0x7fe6a40aef40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000c7f0 tx=0x7fe6a000cbb0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 shutdown_connections 2023-11-08T08:19:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6880682f0 0x7fe68806a7a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6a415c810 0x7fe6a41f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6a406cdc0 0x7fe6a41f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 --2- 172.21.15.116:0/4175947235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a40718e0 0x7fe6a40aef40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 >> 172.21.15.116:0/4175947235 conn(0x7fe6a406c010 msgr2=0x7fe6a406a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 shutdown_connections 2023-11-08T08:19:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.664+0000 7fe6ab9d0700 1 -- 172.21.15.116:0/4175947235 wait complete. 2023-11-08T08:19:41.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:41.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.797+0000 7fec8bd38700 1 Processor -- start 2023-11-08T08:19:41.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.797+0000 7fec8bd38700 1 -- start start 2023-11-08T08:19:41.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.797+0000 7fec8bd38700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 0x7fec84071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.797+0000 7fec8bd38700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec840cfb00 con 0x7fec840cff90 2023-11-08T08:19:41.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec8bd38700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8415aa20 con 0x7fec840b4a30 2023-11-08T08:19:41.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec8bd38700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840cf4d0 con 0x7fec840b4660 2023-11-08T08:19:41.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec892d3700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fec840cff90 0x7fec8406b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48832/0 (socket says 172.21.15.116:48832) 2023-11-08T08:19:41.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec892d3700 1 -- 172.21.15.116:0/248546709 learned_addr learned my addr 172.21.15.116:0/248546709 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:41.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec89ad4700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 0x7fec84071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820352535 0 0) 0x7fec840cfb00 con 0x7fec840cff90 2023-11-08T08:19:41.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6c0035b0 con 0x7fec840cff90 2023-11-08T08:19:41.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818961204 0 0) 0x7fec8415aa20 con 0x7fec840b4a30 2023-11-08T08:19:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec840cfb00 con 0x7fec840b4a30 2023-11-08T08:19:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3617548016 0 0) 0x7fec6c0035b0 con 0x7fec840cff90 2023-11-08T08:19:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 msgr2=0x7fec84071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:41.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 0x7fec84071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 >> v1:172.21.15.129:6789/0 conn(0x7fec840b4a30 legacy=0x7fec8415f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84072550 con 0x7fec840cff90 2023-11-08T08:19:41.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fec78001670 con 0x7fec840cff90 2023-11-08T08:19:41.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 >> v1:172.21.15.116:6789/0 conn(0x7fec840cff90 legacy=0x7fec8406b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 0x7fec6c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec6c004120 0x7fec6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c004d90 0x7fec6c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840cf4d0 con 0x7fec6c004d90 2023-11-08T08:19:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c00bba0 con 0x7fec6c004120 2023-11-08T08:19:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c00bd30 con 0x7fec840b4660 2023-11-08T08:19:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec8a2d5700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c004d90 0x7fec6c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec892d3700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec6c004120 0x7fec6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.798+0000 7fec89ad4700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 0x7fec6c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 -- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 msgr2=0x7fec6c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:41.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec840b4660 0x7fec6c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 -- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec6c004120 msgr2=0x7fec6c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec6c004120 0x7fec6c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 -- 172.21.15.116:0/248546709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0035b0 con 0x7fec6c004d90 2023-11-08T08:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c004d90 0x7fec6c005200 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fec7c001940 tx=0x7fec6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cebf4193a1e0b9a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0052d0 con 0x7fec6c004d90 2023-11-08T08:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c001d70 con 0x7fec6c004d90 2023-11-08T08:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec88ad2700 1 -- 172.21.15.116:0/248546709 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c003680 con 0x7fec6c004d90 2023-11-08T08:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8bd38700 1 -- 172.21.15.116:0/248546709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c004d90 msgr2=0x7fec6c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8bd38700 1 --2- 172.21.15.116:0/248546709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c004d90 0x7fec6c005200 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fec7c001940 tx=0x7fec6c005880 comp rx=0 tx=0).stop 2023-11-08T08:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8a2d5700 1 -- 172.21.15.116:0/248546709 reap_dead start 2023-11-08T08:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8bd38700 1 -- 172.21.15.116:0/248546709 shutdown_connections 2023-11-08T08:19:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8bd38700 1 -- 172.21.15.116:0/248546709 >> 172.21.15.116:0/248546709 conn(0x7fec840b2fc0 msgr2=0x7fec84071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.799+0000 7fec8bd38700 1 -- 172.21.15.116:0/248546709 shutdown_connections 2023-11-08T08:19:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.800+0000 7fec8bd38700 1 -- 172.21.15.116:0/248546709 wait complete. 2023-11-08T08:19:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 Processor -- start 2023-11-08T08:19:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 -- start start 2023-11-08T08:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 0x7fec840afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec8415c550 0x7fec840c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec840aedb0 0x7fec840af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8409bdd0 con 0x7fec840c7e10 2023-11-08T08:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840b0530 con 0x7fec840aedb0 2023-11-08T08:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec8bd38700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c8720 con 0x7fec8415c550 2023-11-08T08:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec89ad4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 0x7fec840afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.801+0000 7fec89ad4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 0x7fec840afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38318/0 (socket says 172.21.15.116:38318) 2023-11-08T08:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 -- 172.21.15.116:0/1338418154 learned_addr learned my addr 172.21.15.116:0/1338418154 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec892d3700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec8415c550 0x7fec840c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec8a2d5700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec840aedb0 0x7fec840af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 -- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec8415c550 msgr2=0x7fec840c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec8415c550 0x7fec840c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 -- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec840aedb0 msgr2=0x7fec840af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec840aedb0 0x7fec840af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 -- 172.21.15.116:0/1338418154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8415f880 con 0x7fec840c7e10 2023-11-08T08:19:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec892d3700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec8415c550 0x7fec840c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec89ad4700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 0x7fec840afff0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fec74011e30 tx=0x7fec7400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7400e040 con 0x7fec840c7e10 2023-11-08T08:19:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7400f040 con 0x7fec840c7e10 2023-11-08T08:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74009c00 con 0x7fec840c7e10 2023-11-08T08:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec840cf790 con 0x7fec840c7e10 2023-11-08T08:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.802+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8415fa10 con 0x7fec840c7e10 2023-11-08T08:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.803+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec840743f0 con 0x7fec840c7e10 2023-11-08T08:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.804+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fec74015650 con 0x7fec840c7e10 2023-11-08T08:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.804+0000 7fec827fc700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fec700682f0 0x7fec7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.804+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fec74086e10 con 0x7fec840c7e10 2023-11-08T08:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.804+0000 7fec892d3700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fec700682f0 0x7fec7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.805+0000 7fec892d3700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fec700682f0 0x7fec7006a7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec780048d0 tx=0x7fec78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.808+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fec74053a60 con 0x7fec840c7e10 2023-11-08T08:19:41.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.989+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec8415f660 con 0x7fec840c7e10 2023-11-08T08:19:41.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.990+0000 7fec827fc700 1 -- 172.21.15.116:0/1338418154 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+14257 (secure 0 0 0) 0x7fec74057080 con 0x7fec840c7e10 2023-11-08T08:19:41.991 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:41.991 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:38.777670+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:41.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fec700682f0 msgr2=0x7fec7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fec700682f0 0x7fec7006a7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec780048d0 tx=0x7fec78009040 comp rx=0 tx=0).stop 2023-11-08T08:19:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 msgr2=0x7fec840afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 0x7fec840afff0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fec74011e30 tx=0x7fec7400f490 comp rx=0 tx=0).stop 2023-11-08T08:19:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 shutdown_connections 2023-11-08T08:19:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fec700682f0 0x7fec7006a7a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec840aedb0 0x7fec840af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec840c7e10 0x7fec840afff0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 --2- 172.21.15.116:0/1338418154 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec8415c550 0x7fec840c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 >> 172.21.15.116:0/1338418154 conn(0x7fec840b2fc0 msgr2=0x7fec8415ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 shutdown_connections 2023-11-08T08:19:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:41.993+0000 7fec8bd38700 1 -- 172.21.15.116:0/1338418154 wait complete. 2023-11-08T08:19:42.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:42.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.128+0000 7f7afffff700 1 Processor -- start 2023-11-08T08:19:42.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.128+0000 7f7afffff700 1 -- start start 2023-11-08T08:19:42.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7b000aff60 0x7f7b000b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afffff700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0015a840 con 0x7f7b000d0aa0 2023-11-08T08:19:42.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afffff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b000ce780 con 0x7f7b000cef40 2023-11-08T08:19:42.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afffff700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0015b5d0 con 0x7f7b000aff60 2023-11-08T08:19:42.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afeffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7b000d0aa0 0x7f7b000d0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48838/0 (socket says 172.21.15.116:48838) 2023-11-08T08:19:42.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afeffd700 1 -- 172.21.15.116:0/1692110056 learned_addr learned my addr 172.21.15.116:0/1692110056 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:42.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.129+0000 7f7afe7fc700 1 --2- 172.21.15.116:0/1692110056 >> v2:172.21.15.129:3300/0 conn(0x7f7b000aff60 0x7f7b000b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242426800 0 0) 0x7f7b0015a840 con 0x7f7b000d0aa0 2023-11-08T08:19:42.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae80035b0 con 0x7f7b000d0aa0 2023-11-08T08:19:42.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996821313 0 0) 0x7f7b000ce780 con 0x7f7b000cef40 2023-11-08T08:19:42.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0015a840 con 0x7f7b000cef40 2023-11-08T08:19:42.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183835953 0 0) 0x7f7ae80035b0 con 0x7f7b000d0aa0 2023-11-08T08:19:42.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afe7fc700 1 -- 172.21.15.116:0/1692110056 >> v1:172.21.15.129:6789/0 conn(0x7f7b000cef40 legacy=0x7f7b000cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afe7fc700 1 -- 172.21.15.116:0/1692110056 >> v1:172.21.15.116:6789/0 conn(0x7f7b000d0aa0 legacy=0x7f7b000d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afe7fc700 1 -- 172.21.15.116:0/1692110056 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b000cea20 con 0x7f7b000aff60 2023-11-08T08:19:42.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afe7fc700 1 --2- 172.21.15.116:0/1692110056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000aff60 0x7f7b000b2590 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009840 tx=0x7f7af0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67785007f9bcecb1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4001670 con 0x7f7b000aff60 2023-11-08T08:19:42.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0004030 con 0x7f7b000aff60 2023-11-08T08:19:42.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.130+0000 7f7afdffb700 1 -- 172.21.15.116:0/1692110056 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af00043c0 con 0x7f7b000aff60 2023-11-08T08:19:42.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 -- 172.21.15.116:0/1692110056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000aff60 msgr2=0x7f7b000b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 --2- 172.21.15.116:0/1692110056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000aff60 0x7f7b000b2590 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009840 tx=0x7f7af0009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:42.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 -- 172.21.15.116:0/1692110056 shutdown_connections 2023-11-08T08:19:42.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 --2- 172.21.15.116:0/1692110056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000aff60 0x7f7b000b2590 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 -- 172.21.15.116:0/1692110056 >> 172.21.15.116:0/1692110056 conn(0x7f7b0006a270 msgr2=0x7f7b0006a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:42.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 -- 172.21.15.116:0/1692110056 shutdown_connections 2023-11-08T08:19:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.131+0000 7f7afffff700 1 -- 172.21.15.116:0/1692110056 wait complete. 2023-11-08T08:19:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.132+0000 7f7afffff700 1 Processor -- start 2023-11-08T08:19:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.132+0000 7f7afffff700 1 -- start start 2023-11-08T08:19:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 0x7f7b00151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000cef40 0x7f7b000c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b000d0aa0 0x7f7b000c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b000c7510 con 0x7f7b000aff60 2023-11-08T08:19:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b001f1790 con 0x7f7b000cef40 2023-11-08T08:19:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b001521d0 con 0x7f7b000d0aa0 2023-11-08T08:19:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 0x7f7b00151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 0x7f7b00151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38324/0 (socket says 172.21.15.116:38324) 2023-11-08T08:19:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 -- 172.21.15.116:0/178463185 learned_addr learned my addr 172.21.15.116:0/178463185 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afe7fc700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000cef40 0x7f7b000c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7aff7fe700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b000d0aa0 0x7f7b000c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 -- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b000d0aa0 msgr2=0x7f7b000c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b000d0aa0 0x7f7b000c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 -- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000cef40 msgr2=0x7f7b000c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000cef40 0x7f7b000c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 -- 172.21.15.116:0/178463185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00094a0 con 0x7f7b000aff60 2023-11-08T08:19:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afeffd700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 0x7f7b00151c90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c6b0 tx=0x7f7af400ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4004c80 con 0x7f7b000aff60 2023-11-08T08:19:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af4004de0 con 0x7f7b000aff60 2023-11-08T08:19:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b000afbd0 con 0x7f7b000aff60 2023-11-08T08:19:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af40072f0 con 0x7f7b000aff60 2023-11-08T08:19:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0015b8b0 con 0x7f7b000aff60 2023-11-08T08:19:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.134+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b000afd60 con 0x7f7b000aff60 2023-11-08T08:19:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.135+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7af401a020 con 0x7f7b000aff60 2023-11-08T08:19:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.135+0000 7f7ae77fe700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7aec0682f0 0x7f7aec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.135+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7af4085640 con 0x7f7b000aff60 2023-11-08T08:19:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.135+0000 7f7afe7fc700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7aec0682f0 0x7f7aec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.136+0000 7f7afe7fc700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7aec0682f0 0x7f7aec06a7a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7b000b0390 tx=0x7f7af0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.140+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7af4052170 con 0x7f7b000aff60 2023-11-08T08:19:42.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.318+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7b000d0910 con 0x7f7b000aff60 2023-11-08T08:19:42.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.318+0000 7f7ae77fe700 1 -- 172.21.15.116:0/178463185 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7af4055790 con 0x7f7b000aff60 2023-11-08T08:19:42.320 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:42.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7aec0682f0 msgr2=0x7f7aec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7aec0682f0 0x7f7aec06a7a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7b000b0390 tx=0x7f7af0008030 comp rx=0 tx=0).stop 2023-11-08T08:19:42.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 msgr2=0x7f7b00151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 0x7f7b00151c90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c6b0 tx=0x7f7af400ca70 comp rx=0 tx=0).stop 2023-11-08T08:19:42.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 shutdown_connections 2023-11-08T08:19:42.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7b000d0aa0 0x7f7b000c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b000aff60 0x7f7b00151c90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7aec0682f0 0x7f7aec06a7a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 --2- 172.21.15.116:0/178463185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b000cef40 0x7f7b000c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 >> 172.21.15.116:0/178463185 conn(0x7f7b0006a270 msgr2=0x7f7b0015d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 shutdown_connections 2023-11-08T08:19:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.320+0000 7f7afffff700 1 -- 172.21.15.116:0/178463185 wait complete. 2023-11-08T08:19:42.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.456+0000 7fd10cfa7700 1 Processor -- start 2023-11-08T08:19:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.456+0000 7fd10cfa7700 1 -- start start 2023-11-08T08:19:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10cfa7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd1081599b0 0x7fd10815d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10cfa7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd1080af0a0 0x7fd10815a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10cfa7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd108159ec0 con 0x7fd1080b1650 2023-11-08T08:19:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10cfa7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd10815d8a0 con 0x7fd1080af0a0 2023-11-08T08:19:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10cfa7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd10815db80 con 0x7fd1081599b0 2023-11-08T08:19:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd106d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd1080af0a0 0x7fd10815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10659c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd1080b1650 0x7fd1080b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36480/0 (socket says 172.21.15.116:36480) 2023-11-08T08:19:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10659c700 1 -- 172.21.15.116:0/2062872916 learned_addr learned my addr 172.21.15.116:0/2062872916 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:42.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd105d9b700 1 --2- 172.21.15.116:0/2062872916 >> v2:172.21.15.129:3300/0 conn(0x7fd1081599b0 0x7fd10815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd106d9d700 1 -- 172.21.15.116:0/2062872916 >> v2:172.21.15.129:3300/0 conn(0x7fd1081599b0 msgr2=0x7fd10815d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:42.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd106d9d700 1 --2- 172.21.15.116:0/2062872916 >> v2:172.21.15.129:3300/0 conn(0x7fd1081599b0 0x7fd10815d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd106d9d700 1 -- 172.21.15.116:0/2062872916 >> v1:172.21.15.116:6789/0 conn(0x7fd1080b1650 legacy=0x7fd1080b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd106d9d700 1 -- 172.21.15.116:0/2062872916 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10815de60 con 0x7fd1080af0a0 2023-11-08T08:19:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd10559a700 1 -- 172.21.15.116:0/2062872916 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996997327 0 0) 0x7fd108159ec0 con 0x7fd1080b1650 2023-11-08T08:19:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.457+0000 7fd105d9b700 1 --2- 172.21.15.116:0/2062872916 >> v2:172.21.15.129:3300/0 conn(0x7fd1081599b0 0x7fd10815d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd106d9d700 1 --2- 172.21.15.116:0/2062872916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080af0a0 0x7fd10815a350 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd10806c730 tx=0x7fd10000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac651cc3932bb83 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10559a700 1 -- 172.21.15.116:0/2062872916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10000e070 con 0x7fd1080af0a0 2023-11-08T08:19:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10559a700 1 -- 172.21.15.116:0/2062872916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10000bbd0 con 0x7fd1080af0a0 2023-11-08T08:19:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10559a700 1 -- 172.21.15.116:0/2062872916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100012430 con 0x7fd1080af0a0 2023-11-08T08:19:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2062872916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080af0a0 msgr2=0x7fd10815a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2062872916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080af0a0 0x7fd10815a350 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd10806c730 tx=0x7fd10000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2062872916 shutdown_connections 2023-11-08T08:19:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2062872916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080af0a0 0x7fd10815a350 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2062872916 >> v2:172.21.15.129:3300/0 conn(0x7fd1081599b0 0x7fd10815d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2062872916 >> 172.21.15.116:0/2062872916 conn(0x7fd10806c010 msgr2=0x7fd1080cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2062872916 shutdown_connections 2023-11-08T08:19:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.458+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2062872916 wait complete. 2023-11-08T08:19:42.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 Processor -- start 2023-11-08T08:19:42.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 -- start start 2023-11-08T08:19:42.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd1080af0a0 0x7fd1080c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1080b1650 0x7fd1080c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 0x7fd10815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1080c3c40 con 0x7fd1081524b0 2023-11-08T08:19:42.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd108157720 con 0x7fd1080b1650 2023-11-08T08:19:42.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.460+0000 7fd10cfa7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1080cb760 con 0x7fd1080af0a0 2023-11-08T08:19:42.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 0x7fd10815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 0x7fd10815bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60036/0 (socket says 172.21.15.116:60036) 2023-11-08T08:19:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 -- 172.21.15.116:0/2211756052 learned_addr learned my addr 172.21.15.116:0/2211756052 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd105d9b700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1080b1650 0x7fd1080c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd10659c700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd1080af0a0 0x7fd1080c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 -- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd1080af0a0 msgr2=0x7fd1080c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd1080af0a0 0x7fd1080c3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 -- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1080b1650 msgr2=0x7fd1080c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1080b1650 0x7fd1080c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 -- 172.21.15.116:0/2211756052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100009a70 con 0x7fd1081524b0 2023-11-08T08:19:42.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd105d9b700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1080b1650 0x7fd1080c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:42.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd106d9d700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 0x7fd10815bc60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd1080708b0 tx=0x7fd100004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100012430 con 0x7fd1081524b0 2023-11-08T08:19:42.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100020070 con 0x7fd1081524b0 2023-11-08T08:19:42.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100004280 con 0x7fd1081524b0 2023-11-08T08:19:42.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10815a890 con 0x7fd1081524b0 2023-11-08T08:19:42.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.461+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10815d590 con 0x7fd1081524b0 2023-11-08T08:19:42.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.462+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10815aa20 con 0x7fd1081524b0 2023-11-08T08:19:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.462+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd10002e070 con 0x7fd1081524b0 2023-11-08T08:19:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.463+0000 7fd0f6ffd700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0e80682f0 0x7fd0e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.463+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd100022070 con 0x7fd1081524b0 2023-11-08T08:19:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.463+0000 7fd10659c700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0e80682f0 0x7fd0e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.463+0000 7fd10659c700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0e80682f0 0x7fd0e806a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc000f60 tx=0x7fd0fc007c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.467+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd100054460 con 0x7fd1081524b0 2023-11-08T08:19:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.647+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1080743f0 con 0x7fd1081524b0 2023-11-08T08:19:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.648+0000 7fd0f6ffd700 1 -- 172.21.15.116:0/2211756052 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd10002c0a0 con 0x7fd1081524b0 2023-11-08T08:19:42.649 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0e80682f0 msgr2=0x7fd0e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0e80682f0 0x7fd0e806a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc000f60 tx=0x7fd0fc007c90 comp rx=0 tx=0).stop 2023-11-08T08:19:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 msgr2=0x7fd10815bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 0x7fd10815bc60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd1080708b0 tx=0x7fd100004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 shutdown_connections 2023-11-08T08:19:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1081524b0 0x7fd10815bc60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd1080af0a0 0x7fd1080c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0e80682f0 0x7fd0e806a7a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 --2- 172.21.15.116:0/2211756052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1080b1650 0x7fd1080c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 >> 172.21.15.116:0/2211756052 conn(0x7fd10806c010 msgr2=0x7fd10806d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 shutdown_connections 2023-11-08T08:19:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.650+0000 7fd10cfa7700 1 -- 172.21.15.116:0/2211756052 wait complete. 2023-11-08T08:19:42.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-08T08:19:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.782+0000 7f10dc133700 1 Processor -- start 2023-11-08T08:19:42.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.782+0000 7f10dc133700 1 -- start start 2023-11-08T08:19:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10dc133700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f10d415cc80 0x7f10d406de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10dc133700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10d406e3a0 0x7f10d4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10dc133700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d40b1810 con 0x7f10d406a270 2023-11-08T08:19:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10dc133700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d40b2490 con 0x7f10d406e3a0 2023-11-08T08:19:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10dc133700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d40b1af0 con 0x7f10d415cc80 2023-11-08T08:19:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10da6d0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10d406e3a0 0x7f10d4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10d9ecf700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f10d406a270 0x7f10d406a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36490/0 (socket says 172.21.15.116:36490) 2023-11-08T08:19:42.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10d9ecf700 1 -- 172.21.15.116:0/1773524778 learned_addr learned my addr 172.21.15.116:0/1773524778 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:42.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10d96ce700 1 --2- 172.21.15.116:0/1773524778 >> v2:172.21.15.129:3300/0 conn(0x7f10d415cc80 0x7f10d406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10da6d0700 1 -- 172.21.15.116:0/1773524778 >> v2:172.21.15.129:3300/0 conn(0x7f10d415cc80 msgr2=0x7f10d406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10da6d0700 1 --2- 172.21.15.116:0/1773524778 >> v2:172.21.15.129:3300/0 conn(0x7f10d415cc80 0x7f10d406de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10da6d0700 1 -- 172.21.15.116:0/1773524778 >> v1:172.21.15.116:6789/0 conn(0x7f10d406a270 legacy=0x7f10d406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10da6d0700 1 -- 172.21.15.116:0/1773524778 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d40b1dd0 con 0x7f10d406e3a0 2023-11-08T08:19:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10da6d0700 1 --2- 172.21.15.116:0/1773524778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406e3a0 0x7f10d4070d20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f10d406c6c0 tx=0x7f10cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5871c3d1c852e9e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.783+0000 7f10d8ecd700 1 -- 172.21.15.116:0/1773524778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00e070 con 0x7f10d406e3a0 2023-11-08T08:19:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10d8ecd700 1 -- 172.21.15.116:0/1773524778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc00bbd0 con 0x7f10d406e3a0 2023-11-08T08:19:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10d8ecd700 1 -- 172.21.15.116:0/1773524778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc012430 con 0x7f10d406e3a0 2023-11-08T08:19:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 -- 172.21.15.116:0/1773524778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406e3a0 msgr2=0x7f10d4070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 --2- 172.21.15.116:0/1773524778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406e3a0 0x7f10d4070d20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f10d406c6c0 tx=0x7f10cc00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 -- 172.21.15.116:0/1773524778 shutdown_connections 2023-11-08T08:19:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 --2- 172.21.15.116:0/1773524778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406e3a0 0x7f10d4070d20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 --2- 172.21.15.116:0/1773524778 >> v2:172.21.15.129:3300/0 conn(0x7f10d415cc80 0x7f10d406de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 -- 172.21.15.116:0/1773524778 >> 172.21.15.116:0/1773524778 conn(0x7f10d406c010 msgr2=0x7f10d40cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 -- 172.21.15.116:0/1773524778 shutdown_connections 2023-11-08T08:19:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.784+0000 7f10dc133700 1 -- 172.21.15.116:0/1773524778 wait complete. 2023-11-08T08:19:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.785+0000 7f10dc133700 1 Processor -- start 2023-11-08T08:19:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.785+0000 7f10dc133700 1 -- start start 2023-11-08T08:19:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10dc133700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 0x7f10d40c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10dc133700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10d406e3a0 0x7f10d40c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10dc133700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d415cc80 0x7f10d40b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10dc133700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40c6460 con 0x7f10d406a270 2023-11-08T08:19:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10dc133700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40c6180 con 0x7f10d415cc80 2023-11-08T08:19:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10dc133700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d41e5ba0 con 0x7f10d406e3a0 2023-11-08T08:19:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 0x7f10d40c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 0x7f10d40c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60048/0 (socket says 172.21.15.116:60048) 2023-11-08T08:19:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 -- 172.21.15.116:0/3844756230 learned_addr learned my addr 172.21.15.116:0/3844756230 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:42.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 -- 172.21.15.116:0/3844756230 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10d406e3a0 msgr2=0x7f10d40c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10d406e3a0 0x7f10d40c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 -- 172.21.15.116:0/3844756230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d415cc80 msgr2=0x7f10d40b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d415cc80 0x7f10d40b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.786+0000 7f10d9ecf700 1 -- 172.21.15.116:0/3844756230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc009a70 con 0x7f10d406a270 2023-11-08T08:19:42.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.787+0000 7f10d9ecf700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 0x7f10d40c5ba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f10d000c480 tx=0x7f10d000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.787+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0005d70 con 0x7f10d406a270 2023-11-08T08:19:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.787+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d00070d0 con 0x7f10d406a270 2023-11-08T08:19:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.787+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d406e7c0 con 0x7f10d406a270 2023-11-08T08:19:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.787+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0007440 con 0x7f10d406a270 2023-11-08T08:19:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.787+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d40b20b0 con 0x7f10d406a270 2023-11-08T08:19:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.788+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d406e950 con 0x7f10d406a270 2023-11-08T08:19:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.788+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10d0019030 con 0x7f10d406a270 2023-11-08T08:19:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.788+0000 7f10ca7fc700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10b80682f0 0x7f10b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.789+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f10d0084c20 con 0x7f10d406a270 2023-11-08T08:19:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.789+0000 7f10d96ce700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10b80682f0 0x7f10b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.789+0000 7f10d96ce700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10b80682f0 0x7f10b806a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f10d40cc250 tx=0x7f10c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.793+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10d004b5e0 con 0x7f10d406a270 2023-11-08T08:19:42.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.974+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f10d40743f0 con 0x7f10d406a270 2023-11-08T08:19:42.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.975+0000 7f10ca7fc700 1 -- 172.21.15.116:0/3844756230 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) v1 ==== 84+0+126 (secure 0 0 0) 0x7f10d0017090 con 0x7f10d406a270 2023-11-08T08:19:42.977 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:42.978 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-08T08:19:42.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:42.978 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-08T08:19:42.979 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": "rbd" 2023-11-08T08:19:42.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:42.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:19:42.980 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 2, 2023-11-08T08:19:42.980 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-08T08:19:42.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:42.982 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-08T08:19:42.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10b80682f0 msgr2=0x7f10b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10b80682f0 0x7f10b806a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f10d40cc250 tx=0x7f10c4008040 comp rx=0 tx=0).stop 2023-11-08T08:19:42.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 msgr2=0x7f10d40c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:42.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 0x7f10d40c5ba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f10d000c480 tx=0x7f10d000c840 comp rx=0 tx=0).stop 2023-11-08T08:19:42.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 shutdown_connections 2023-11-08T08:19:42.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f10b80682f0 0x7f10b806a7a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d415cc80 0x7f10d40b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10d406a270 0x7f10d40c5ba0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 --2- 172.21.15.116:0/3844756230 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f10d406e3a0 0x7f10d40c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:42.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 >> 172.21.15.116:0/3844756230 conn(0x7f10d406c010 msgr2=0x7f10d40b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:42.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 shutdown_connections 2023-11-08T08:19:42.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:42.977+0000 7f10dc133700 1 -- 172.21.15.116:0/3844756230 wait complete. 2023-11-08T08:19:42.990 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-08T08:19:42.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-08T08:19:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.112+0000 7f0ebffec700 1 Processor -- start 2023-11-08T08:19:43.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 -- start start 2023-11-08T08:19:43.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0eb815b090 0x7f0eb815b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0eb80c9bd0 0x7f0eb806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb815b9a0 con 0x7f0eb815b090 2023-11-08T08:19:43.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb806cd50 con 0x7f0eb80c9bd0 2023-11-08T08:19:43.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebffec700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb8071500 con 0x7f0eb806c390 2023-11-08T08:19:43.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.113+0000 7f0ebdd88700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0eb815b090 0x7f0eb815b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0eb815b090 0x7f0eb815b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60056/0 (socket says 172.21.15.116:60056) 2023-11-08T08:19:43.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 -- 172.21.15.116:0/1139251217 learned_addr learned my addr 172.21.15.116:0/1139251217 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:43.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebd587700 1 --2- 172.21.15.116:0/1139251217 >> v2:172.21.15.129:3300/0 conn(0x7f0eb80c9bd0 0x7f0eb806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebe589700 1 --2- 172.21.15.116:0/1139251217 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 -- 172.21.15.116:0/1139251217 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 msgr2=0x7f0eb806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:43.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 --2- 172.21.15.116:0/1139251217 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 -- 172.21.15.116:0/1139251217 >> v2:172.21.15.129:3300/0 conn(0x7f0eb80c9bd0 msgr2=0x7f0eb806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:43.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 --2- 172.21.15.116:0/1139251217 >> v2:172.21.15.129:3300/0 conn(0x7f0eb80c9bd0 0x7f0eb806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 -- 172.21.15.116:0/1139251217 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb806aaa0 con 0x7f0eb815b090 2023-11-08T08:19:43.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebdd88700 1 --2- 172.21.15.116:0/1139251217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815b460 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0eb80c2e70 tx=0x7f0ea8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5906ee5f47fd2806 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:43.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebcd86700 1 -- 172.21.15.116:0/1139251217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800b870 con 0x7f0eb815b090 2023-11-08T08:19:43.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebcd86700 1 -- 172.21.15.116:0/1139251217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea800b9d0 con 0x7f0eb815b090 2023-11-08T08:19:43.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.114+0000 7f0ebcd86700 1 -- 172.21.15.116:0/1139251217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800bd60 con 0x7f0eb815b090 2023-11-08T08:19:43.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 -- 172.21.15.116:0/1139251217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 msgr2=0x7f0eb815b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 --2- 172.21.15.116:0/1139251217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815b460 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0eb80c2e70 tx=0x7f0ea8009bd0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 -- 172.21.15.116:0/1139251217 shutdown_connections 2023-11-08T08:19:43.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 --2- 172.21.15.116:0/1139251217 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb806c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 --2- 172.21.15.116:0/1139251217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815b460 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 --2- 172.21.15.116:0/1139251217 >> v2:172.21.15.129:3300/0 conn(0x7f0eb80c9bd0 0x7f0eb806a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 -- 172.21.15.116:0/1139251217 >> 172.21.15.116:0/1139251217 conn(0x7f0eb80cf370 msgr2=0x7f0eb80cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:43.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 -- 172.21.15.116:0/1139251217 shutdown_connections 2023-11-08T08:19:43.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.115+0000 7f0ebffec700 1 -- 172.21.15.116:0/1139251217 wait complete. 2023-11-08T08:19:43.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.116+0000 7f0ebffec700 1 Processor -- start 2023-11-08T08:19:43.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.116+0000 7f0ebffec700 1 -- start start 2023-11-08T08:19:43.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb80c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eb80c9bd0 0x7f0eb815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b4930 con 0x7f0eb815b090 2023-11-08T08:19:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80c7390 con 0x7f0eb80c9bd0 2023-11-08T08:19:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80c7670 con 0x7f0eb806c390 2023-11-08T08:19:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60058/0 (socket says 172.21.15.116:60058) 2023-11-08T08:19:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 -- 172.21.15.116:0/2275257918 learned_addr learned my addr 172.21.15.116:0/2275257918 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebd587700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eb80c9bd0 0x7f0eb815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebdd88700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb80c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 -- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 msgr2=0x7f0eb80c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb80c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 -- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eb80c9bd0 msgr2=0x7f0eb815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eb80c9bd0 0x7f0eb815c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 -- 172.21.15.116:0/2275257918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea8009550 con 0x7f0eb815b090 2023-11-08T08:19:43.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebe589700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815c650 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0007d20 tx=0x7f0eb000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:43.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb000ed30 con 0x7f0eb815b090 2023-11-08T08:19:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb0007e70 con 0x7f0eb815b090 2023-11-08T08:19:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb806b060 con 0x7f0eb815b090 2023-11-08T08:19:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0014640 con 0x7f0eb815b090 2023-11-08T08:19:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.117+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb806ad80 con 0x7f0eb815b090 2023-11-08T08:19:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.118+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb806b340 con 0x7f0eb815b090 2023-11-08T08:19:43.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.119+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0eb0012050 con 0x7f0eb815b090 2023-11-08T08:19:43.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.119+0000 7f0eae7fc700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0e9c0682f0 0x7f0e9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.119+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0eb0085e90 con 0x7f0eb815b090 2023-11-08T08:19:43.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.119+0000 7f0ebdd88700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0e9c0682f0 0x7f0e9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.119+0000 7f0ebdd88700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0e9c0682f0 0x7f0e9c06a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8000c00 tx=0x7f0ea8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:43.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.123+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0eb00529b0 con 0x7f0eb815b090 2023-11-08T08:19:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.304+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0eb80c71f0 con 0x7f0eb815b090 2023-11-08T08:19:43.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.668+0000 7f0eae7fc700 1 -- 172.21.15.116:0/2275257918 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0eb80c71f0 con 0x7f0eb815b090 2023-11-08T08:19:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0e9c0682f0 msgr2=0x7f0e9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0e9c0682f0 0x7f0e9c06a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8000c00 tx=0x7f0ea8008030 comp rx=0 tx=0).stop 2023-11-08T08:19:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 msgr2=0x7f0eb815c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815c650 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0007d20 tx=0x7f0eb000e3c0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 shutdown_connections 2023-11-08T08:19:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb815b090 0x7f0eb815c650 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eb806c390 0x7f0eb80c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0e9c0682f0 0x7f0e9c06a7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 --2- 172.21.15.116:0/2275257918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eb80c9bd0 0x7f0eb815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 >> 172.21.15.116:0/2275257918 conn(0x7f0eb80cf370 msgr2=0x7f0eb80cf740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:43.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 shutdown_connections 2023-11-08T08:19:43.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.672+0000 7f0ebffec700 1 -- 172.21.15.116:0/2275257918 wait complete. 2023-11-08T08:19:43.675 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-08T08:19:43.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-08T08:19:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.814+0000 7fd03c614700 1 Processor -- start 2023-11-08T08:19:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.814+0000 7fd03c614700 1 -- start start 2023-11-08T08:19:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03c614700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd03406a980 0x7fd0340ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03c614700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd034071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03c614700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03415cc10 con 0x7fd0340cf710 2023-11-08T08:19:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03c614700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03406c9b0 con 0x7fd03406a980 2023-11-08T08:19:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03c614700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03406cc90 con 0x7fd034071890 2023-11-08T08:19:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03a3b0700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd0340cf710 0x7fd03406a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36502/0 (socket says 172.21.15.116:36502) 2023-11-08T08:19:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03a3b0700 1 -- 172.21.15.116:0/1073645533 learned_addr learned my addr 172.21.15.116:0/1073645533 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:43.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077826506 0 0) 0x7fd03415cc10 con 0x7fd0340cf710 2023-11-08T08:19:43.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0200035b0 con 0x7fd0340cf710 2023-11-08T08:19:43.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd039baf700 1 --2- 172.21.15.116:0/1073645533 >> v2:172.21.15.129:3300/0 conn(0x7fd03406a980 0x7fd0340ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.815+0000 7fd03abb1700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd034071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007918122 0 0) 0x7fd0200035b0 con 0x7fd0340cf710 2023-11-08T08:19:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 msgr2=0x7fd034071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd034071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 >> v2:172.21.15.129:3300/0 conn(0x7fd03406a980 msgr2=0x7fd0340ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 --2- 172.21.15.116:0/1073645533 >> v2:172.21.15.129:3300/0 conn(0x7fd03406a980 0x7fd0340ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03406cf70 con 0x7fd0340cf710 2023-11-08T08:19:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fd024001670 con 0x7fd0340cf710 2023-11-08T08:19:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd034071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd039baf700 1 --2- 172.21.15.116:0/1073645533 >> v2:172.21.15.129:3300/0 conn(0x7fd03406a980 0x7fd0340ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 >> v1:172.21.15.116:6789/0 conn(0x7fd0340cf710 legacy=0x7fd03406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd020003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020003ff0 0x7fd020004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd020004d90 0x7fd020005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03406c9b0 con 0x7fd020004d90 2023-11-08T08:19:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03406cc90 con 0x7fd020003ff0 2023-11-08T08:19:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02000bb70 con 0x7fd034071890 2023-11-08T08:19:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd020004d90 0x7fd020005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03a3b0700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd020003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd039baf700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020003ff0 0x7fd020004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 -- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 msgr2=0x7fd020003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:43.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd034071890 0x7fd020003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 -- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020003ff0 msgr2=0x7fd020004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:43.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020003ff0 0x7fd020004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 -- 172.21.15.116:0/1073645533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200035b0 con 0x7fd020004d90 2023-11-08T08:19:43.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd03abb1700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd020004d90 0x7fd020005200 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0073b0 tx=0x7fd020005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42af852c97e38f5d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:43.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c007e70 con 0x7fd020004d90 2023-11-08T08:19:43.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c014400 con 0x7fd020004d90 2023-11-08T08:19:43.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.816+0000 7fd0393ae700 1 -- 172.21.15.116:0/1073645533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c0147b0 con 0x7fd020004d90 2023-11-08T08:19:43.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03c614700 1 -- 172.21.15.116:0/1073645533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd020004d90 msgr2=0x7fd020005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03c614700 1 --2- 172.21.15.116:0/1073645533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd020004d90 0x7fd020005200 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0073b0 tx=0x7fd020005880 comp rx=0 tx=0).stop 2023-11-08T08:19:43.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03abb1700 1 -- 172.21.15.116:0/1073645533 reap_dead start 2023-11-08T08:19:43.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03c614700 1 -- 172.21.15.116:0/1073645533 shutdown_connections 2023-11-08T08:19:43.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03c614700 1 -- 172.21.15.116:0/1073645533 >> 172.21.15.116:0/1073645533 conn(0x7fd03406c010 msgr2=0x7fd0340cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03c614700 1 -- 172.21.15.116:0/1073645533 shutdown_connections 2023-11-08T08:19:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.817+0000 7fd03c614700 1 -- 172.21.15.116:0/1073645533 wait complete. 2023-11-08T08:19:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.818+0000 7fd03c614700 1 Processor -- start 2023-11-08T08:19:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.818+0000 7fd03c614700 1 -- start start 2023-11-08T08:19:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03c614700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020004d90 0x7fd0340b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03c614700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0340c6d50 0x7fd0340c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03c614700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 0x7fd03415f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03c614700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03415fcb0 con 0x7fd03415f300 2023-11-08T08:19:43.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03c614700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0341f5740 con 0x7fd020004d90 2023-11-08T08:19:43.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03c614700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0341f5a20 con 0x7fd0340c6d50 2023-11-08T08:19:43.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 0x7fd03415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 0x7fd03415f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60080/0 (socket says 172.21.15.116:60080) 2023-11-08T08:19:43.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 -- 172.21.15.116:0/804259551 learned_addr learned my addr 172.21.15.116:0/804259551 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:43.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd039baf700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0340c6d50 0x7fd0340c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03a3b0700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020004d90 0x7fd0340b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 -- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0340c6d50 msgr2=0x7fd0340c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0340c6d50 0x7fd0340c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 -- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020004d90 msgr2=0x7fd0340b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020004d90 0x7fd0340b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 -- 172.21.15.116:0/804259551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03406d530 con 0x7fd03415f300 2023-11-08T08:19:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.819+0000 7fd03abb1700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 0x7fd03415f770 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd02000bac0 tx=0x7fd02c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.820+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c007000 con 0x7fd03415f300 2023-11-08T08:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.820+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c009470 con 0x7fd03415f300 2023-11-08T08:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.820+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03406d250 con 0x7fd03415f300 2023-11-08T08:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.820+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c01e650 con 0x7fd03415f300 2023-11-08T08:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.820+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd034071220 con 0x7fd03415f300 2023-11-08T08:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.821+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0341f6500 con 0x7fd03415f300 2023-11-08T08:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.821+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd02c012070 con 0x7fd03415f300 2023-11-08T08:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.821+0000 7fd02affd700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0180682f0 0x7fd01806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.821+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fd02c086e00 con 0x7fd03415f300 2023-11-08T08:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.821+0000 7fd03a3b0700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0180682f0 0x7fd01806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.822+0000 7fd03a3b0700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0180682f0 0x7fd01806a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd024001630 tx=0x7fd024009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:43.826+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0341f6500 con 0x7fd03415f300 2023-11-08T08:19:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.006+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd0341f6500 con 0x7fd03415f300 2023-11-08T08:19:44.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.680+0000 7fd02affd700 1 -- 172.21.15.116:0/804259551 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v30) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd02c0537e0 con 0x7fd03415f300 2023-11-08T08:19:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0180682f0 msgr2=0x7fd01806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0180682f0 0x7fd01806a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd024001630 tx=0x7fd024009040 comp rx=0 tx=0).stop 2023-11-08T08:19:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 msgr2=0x7fd03415f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 0x7fd03415f770 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd02000bac0 tx=0x7fd02c004b40 comp rx=0 tx=0).stop 2023-11-08T08:19:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 shutdown_connections 2023-11-08T08:19:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd03415f300 0x7fd03415f770 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0180682f0 0x7fd01806a7a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd020004d90 0x7fd0340b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 --2- 172.21.15.116:0/804259551 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0340c6d50 0x7fd0340c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.687+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 >> 172.21.15.116:0/804259551 conn(0x7fd03406c010 msgr2=0x7fd0340ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.688+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 shutdown_connections 2023-11-08T08:19:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.688+0000 7fd03c614700 1 -- 172.21.15.116:0/804259551 wait complete. 2023-11-08T08:19:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-08T08:19:44.706 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-08T08:19:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.830+0000 7fa25b9d7700 1 Processor -- start 2023-11-08T08:19:44.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.831+0000 7fa25b9d7700 1 -- start start 2023-11-08T08:19:44.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.831+0000 7fa25b9d7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2540cec00 0x7fa254065fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:44.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.831+0000 7fa25b9d7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa2540cac90 0x7fa2540cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:44.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.831+0000 7fa25b9d7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25415ceb0 con 0x7fa254160300 2023-11-08T08:19:44.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.831+0000 7fa25b9d7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2540cb650 con 0x7fa2540cec00 2023-11-08T08:19:44.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.831+0000 7fa25b9d7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2541608c0 con 0x7fa2540cac90 2023-11-08T08:19:44.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259773700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2540cec00 0x7fa254065fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:44.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa258f72700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa254160300 0x7fa25415d340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36504/0 (socket says 172.21.15.116:36504) 2023-11-08T08:19:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa258f72700 1 -- 172.21.15.116:0/704100504 learned_addr learned my addr 172.21.15.116:0/704100504 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259f74700 1 --2- 172.21.15.116:0/704100504 >> v2:172.21.15.129:3300/0 conn(0x7fa2540cac90 0x7fa2540cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259773700 1 -- 172.21.15.116:0/704100504 >> v2:172.21.15.129:3300/0 conn(0x7fa2540cac90 msgr2=0x7fa2540cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259773700 1 --2- 172.21.15.116:0/704100504 >> v2:172.21.15.129:3300/0 conn(0x7fa2540cac90 0x7fa2540cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259773700 1 -- 172.21.15.116:0/704100504 >> v1:172.21.15.116:6789/0 conn(0x7fa254160300 legacy=0x7fa25415d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259773700 1 -- 172.21.15.116:0/704100504 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254160ba0 con 0x7fa2540cec00 2023-11-08T08:19:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa24bfff700 1 -- 172.21.15.116:0/704100504 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835043743 0 0) 0x7fa25415ceb0 con 0x7fa254160300 2023-11-08T08:19:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259f74700 1 --2- 172.21.15.116:0/704100504 >> v2:172.21.15.129:3300/0 conn(0x7fa2540cac90 0x7fa2540cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa259773700 1 --2- 172.21.15.116:0/704100504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2540cec00 0x7fa254065fd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa25406bae0 tx=0x7fa244009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb95d337e4908b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa24bfff700 1 -- 172.21.15.116:0/704100504 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244011830 con 0x7fa2540cec00 2023-11-08T08:19:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa24bfff700 1 -- 172.21.15.116:0/704100504 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244011990 con 0x7fa2540cec00 2023-11-08T08:19:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.832+0000 7fa24bfff700 1 -- 172.21.15.116:0/704100504 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244011d20 con 0x7fa2540cec00 2023-11-08T08:19:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 -- 172.21.15.116:0/704100504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2540cec00 msgr2=0x7fa254065fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/704100504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2540cec00 0x7fa254065fd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa25406bae0 tx=0x7fa244009b20 comp rx=0 tx=0).stop 2023-11-08T08:19:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 -- 172.21.15.116:0/704100504 shutdown_connections 2023-11-08T08:19:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/704100504 >> v2:172.21.15.129:3300/0 conn(0x7fa2540cac90 0x7fa2540cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/704100504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2540cec00 0x7fa254065fd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 -- 172.21.15.116:0/704100504 >> 172.21.15.116:0/704100504 conn(0x7fa2540cf400 msgr2=0x7fa2540cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:44.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 -- 172.21.15.116:0/704100504 shutdown_connections 2023-11-08T08:19:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.833+0000 7fa25b9d7700 1 -- 172.21.15.116:0/704100504 wait complete. 2023-11-08T08:19:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.834+0000 7fa25b9d7700 1 Processor -- start 2023-11-08T08:19:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.834+0000 7fa25b9d7700 1 -- start start 2023-11-08T08:19:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa25b9d7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2540cac90 0x7fa2541564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa25b9d7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2540cec00 0x7fa2540b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa25b9d7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 0x7fa2540712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa25b9d7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2541e75d0 con 0x7fa254160300 2023-11-08T08:19:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa25b9d7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2541e7820 con 0x7fa2540cec00 2023-11-08T08:19:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa25b9d7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2541e7b00 con 0x7fa2540cac90 2023-11-08T08:19:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa259f74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 0x7fa2540712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa259f74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 0x7fa2540712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60106/0 (socket says 172.21.15.116:60106) 2023-11-08T08:19:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.835+0000 7fa259f74700 1 -- 172.21.15.116:0/1971077927 learned_addr learned my addr 172.21.15.116:0/1971077927 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa259f74700 1 -- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2540cac90 msgr2=0x7fa2541564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa259f74700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2540cac90 0x7fa2541564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa259f74700 1 -- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2540cec00 msgr2=0x7fa2540b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa258f72700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2540cec00 0x7fa2540b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa259f74700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2540cec00 0x7fa2540b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa259f74700 1 -- 172.21.15.116:0/1971077927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440094a0 con 0x7fa254160300 2023-11-08T08:19:44.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa258f72700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2540cec00 0x7fa2540b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:44.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa259f74700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 0x7fa2540712c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00cef0 tx=0x7fa24c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:44.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00ae50 con 0x7fa254160300 2023-11-08T08:19:44.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c015440 con 0x7fa254160300 2023-11-08T08:19:44.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0157f0 con 0x7fa254160300 2023-11-08T08:19:44.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa254161160 con 0x7fa254160300 2023-11-08T08:19:44.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.836+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa254160e80 con 0x7fa254160300 2023-11-08T08:19:44.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.837+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa254066510 con 0x7fa254160300 2023-11-08T08:19:44.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.837+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa24c01c030 con 0x7fa254160300 2023-11-08T08:19:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.838+0000 7fa249ffb700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2380682f0 0x7fa23806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.838+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fa24c0860d0 con 0x7fa254160300 2023-11-08T08:19:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.838+0000 7fa259773700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2380682f0 0x7fa23806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.838+0000 7fa259773700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2380682f0 0x7fa23806a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa244000c00 tx=0x7fa244010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:44.842+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa24c052990 con 0x7fa254160300 2023-11-08T08:19:45.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.063+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa2540743f0 con 0x7fa254160300 2023-11-08T08:19:45.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.070+0000 7fa249ffb700 1 -- 172.21.15.116:0/1971077927 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa24c055fb0 con 0x7fa254160300 2023-11-08T08:19:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.074+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2380682f0 msgr2=0x7fa23806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.074+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2380682f0 0x7fa23806a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa244000c00 tx=0x7fa244010000 comp rx=0 tx=0).stop 2023-11-08T08:19:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.074+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 msgr2=0x7fa2540712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.074+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 0x7fa2540712c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00cef0 tx=0x7fa24c00a340 comp rx=0 tx=0).stop 2023-11-08T08:19:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.076+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 shutdown_connections 2023-11-08T08:19:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.076+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa254160300 0x7fa2540712c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.076+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa2540cac90 0x7fa2541564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.076+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa2380682f0 0x7fa23806a7a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.076+0000 7fa25b9d7700 1 --2- 172.21.15.116:0/1971077927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2540cec00 0x7fa2540b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.076+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 >> 172.21.15.116:0/1971077927 conn(0x7fa2540cf400 msgr2=0x7fa2540aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.084+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 shutdown_connections 2023-11-08T08:19:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.085+0000 7fa25b9d7700 1 -- 172.21.15.116:0/1971077927 wait complete. 2023-11-08T08:19:45.090 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 5 and data pool 6 2023-11-08T08:19:45.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:45.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.225+0000 7f92bedb1700 1 Processor -- start 2023-11-08T08:19:45.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.225+0000 7f92bedb1700 1 -- start start 2023-11-08T08:19:45.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92bedb1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f92b806a270 0x7f92b806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92bedb1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b8151a50 con 0x7f92b8070d90 2023-11-08T08:19:45.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92bedb1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b815b440 con 0x7f92b815e0b0 2023-11-08T08:19:45.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92bedb1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b806ab80 con 0x7f92b806a270 2023-11-08T08:19:45.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b7fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f92b8070d90 0x7f92b80711e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36506/0 (socket says 172.21.15.116:36506) 2023-11-08T08:19:45.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b7fff700 1 -- 172.21.15.116:0/3147641772 learned_addr learned my addr 172.21.15.116:0/3147641772 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:45.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112303210 0 0) 0x7f92b8151a50 con 0x7f92b8070d90 2023-11-08T08:19:45.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a80035b0 con 0x7f92b8070d90 2023-11-08T08:19:45.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92bcb4d700 1 --2- 172.21.15.116:0/3147641772 >> v2:172.21.15.129:3300/0 conn(0x7f92b806a270 0x7f92b806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935112188 0 0) 0x7f92a80035b0 con 0x7f92b8070d90 2023-11-08T08:19:45.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 >> v2:172.21.15.129:3300/0 conn(0x7f92b806a270 msgr2=0x7f92b806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 --2- 172.21.15.116:0/3147641772 >> v2:172.21.15.129:3300/0 conn(0x7f92b806a270 0x7f92b806a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 >> v1:172.21.15.129:6789/0 conn(0x7f92b815e0b0 legacy=0x7f92b815e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8071ac0 con 0x7f92b8070d90 2023-11-08T08:19:45.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f92a4001670 con 0x7f92b8070d90 2023-11-08T08:19:45.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 >> v1:172.21.15.116:6789/0 conn(0x7f92b8070d90 legacy=0x7f92b80711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92a8003a00 0x7f92a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92a8004850 0x7f92a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.226+0000 7f92b77fe700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b806ab80 con 0x7f92a8005200 2023-11-08T08:19:45.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a800c080 con 0x7f92a8004850 2023-11-08T08:19:45.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a800c240 con 0x7f92a8003a00 2023-11-08T08:19:45.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b7fff700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92a8004850 0x7f92a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bcb4d700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92a8003a00 0x7f92a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 -- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92a8003a00 msgr2=0x7f92a8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92a8003a00 0x7f92a8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 -- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92a8004850 msgr2=0x7f92a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92a8004850 0x7f92a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 -- 172.21.15.116:0/3147641772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a80035b0 con 0x7f92a8005200 2023-11-08T08:19:45.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 -- 172.21.15.116:0/3147641772 reap_dead start 2023-11-08T08:19:45.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92bd34e700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92a800ba10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f92b000c190 tx=0x7f92a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6e6e29864c6771 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0001db0 con 0x7f92a8005200 2023-11-08T08:19:45.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b000ccd0 con 0x7f92a8005200 2023-11-08T08:19:45.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.227+0000 7f92b77fe700 1 -- 172.21.15.116:0/3147641772 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0003c40 con 0x7f92a8005200 2023-11-08T08:19:45.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 -- 172.21.15.116:0/3147641772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 msgr2=0x7f92a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92a800ba10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f92b000c190 tx=0x7f92a8005880 comp rx=0 tx=0).stop 2023-11-08T08:19:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 -- 172.21.15.116:0/3147641772 shutdown_connections 2023-11-08T08:19:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 --2- 172.21.15.116:0/3147641772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92a800ba10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 -- 172.21.15.116:0/3147641772 >> 172.21.15.116:0/3147641772 conn(0x7f92b806c010 msgr2=0x7f92b80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 -- 172.21.15.116:0/3147641772 shutdown_connections 2023-11-08T08:19:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.228+0000 7f92bedb1700 1 -- 172.21.15.116:0/3147641772 wait complete. 2023-11-08T08:19:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.229+0000 7f92bedb1700 1 Processor -- start 2023-11-08T08:19:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.229+0000 7f92bedb1700 1 -- start start 2023-11-08T08:19:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bedb1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92b80ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bedb1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b8148d80 0x7f92b81f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bedb1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b80cf0c0 0x7f92b8153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bedb1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80ce950 con 0x7f92a8005200 2023-11-08T08:19:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bedb1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b81f69e0 con 0x7f92b8148d80 2023-11-08T08:19:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bedb1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80c6dc0 con 0x7f92b80cf0c0 2023-11-08T08:19:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92b80ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92b80ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60126/0 (socket says 172.21.15.116:60126) 2023-11-08T08:19:45.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 -- 172.21.15.116:0/567330889 learned_addr learned my addr 172.21.15.116:0/567330889 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:45.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92b7fff700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b8148d80 0x7f92b81f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bd34e700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b80cf0c0 0x7f92b8153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 -- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b80cf0c0 msgr2=0x7f92b8153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b80cf0c0 0x7f92b8153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 -- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b8148d80 msgr2=0x7f92b81f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b8148d80 0x7f92b81f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 -- 172.21.15.116:0/567330889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b815b0d0 con 0x7f92a8005200 2023-11-08T08:19:45.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92b7fff700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b8148d80 0x7f92b81f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:45.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92bcb4d700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92b80ce410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0119a0 tx=0x7f92ac011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.230+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac00f8e0 con 0x7f92a8005200 2023-11-08T08:19:45.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.231+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac00fa40 con 0x7f92a8005200 2023-11-08T08:19:45.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.231+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b815adf0 con 0x7f92a8005200 2023-11-08T08:19:45.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.231+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac00fdd0 con 0x7f92a8005200 2023-11-08T08:19:45.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.231+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b815aac0 con 0x7f92a8005200 2023-11-08T08:19:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.232+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b806af90 con 0x7f92a8005200 2023-11-08T08:19:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.232+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92ac014440 con 0x7f92a8005200 2023-11-08T08:19:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.232+0000 7f92b57fa700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f929c068320 0x7f929c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.232+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f92ac086420 con 0x7f92a8005200 2023-11-08T08:19:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.232+0000 7f92b7fff700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f929c068320 0x7f929c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.233+0000 7f92b7fff700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f929c068320 0x7f929c06a7d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92a4001050 tx=0x7f92a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.236+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92ac052c20 con 0x7f92a8005200 2023-11-08T08:19:45.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.419+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b80743f0 con 0x7f92a8005200 2023-11-08T08:19:45.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.420+0000 7f92b57fa700 1 -- 172.21.15.116:0/567330889 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+17982 (secure 0 0 0) 0x7f92ac056240 con 0x7f92a8005200 2023-11-08T08:19:45.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:45.422 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:45.064246+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:45.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f929c068320 msgr2=0x7f929c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f929c068320 0x7f929c06a7d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92a4001050 tx=0x7f92a4009040 comp rx=0 tx=0).stop 2023-11-08T08:19:45.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 msgr2=0x7f92b80ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92b80ce410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0119a0 tx=0x7f92ac011d60 comp rx=0 tx=0).stop 2023-11-08T08:19:45.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 shutdown_connections 2023-11-08T08:19:45.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f92b80cf0c0 0x7f92b8153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92a8005200 0x7f92b80ce410 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f929c068320 0x7f929c06a7d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 --2- 172.21.15.116:0/567330889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92b8148d80 0x7f92b81f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 >> 172.21.15.116:0/567330889 conn(0x7f92b806c010 msgr2=0x7f92b80b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:45.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 shutdown_connections 2023-11-08T08:19:45.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.422+0000 7f92bedb1700 1 -- 172.21.15.116:0/567330889 wait complete. 2023-11-08T08:19:45.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:45.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.555+0000 7fbf9023f700 1 Processor -- start 2023-11-08T08:19:45.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.555+0000 7fbf9023f700 1 -- start start 2023-11-08T08:19:45.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf9023f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbf8806b7b0 0x7fbf88072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf9023f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf880ad1f0 0x7fbf880ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf9023f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf880726c0 con 0x7fbf880b0880 2023-11-08T08:19:45.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf9023f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8806bc50 con 0x7fbf880ad1f0 2023-11-08T08:19:45.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf9023f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf880adb50 con 0x7fbf8806b7b0 2023-11-08T08:19:45.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf880ad1f0 0x7fbf880ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf880ad1f0 0x7fbf880ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60136/0 (socket says 172.21.15.116:60136) 2023-11-08T08:19:45.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 -- 172.21.15.116:0/866859394 learned_addr learned my addr 172.21.15.116:0/866859394 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:45.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8d7da700 1 --2- 172.21.15.116:0/866859394 >> v2:172.21.15.129:3300/0 conn(0x7fbf8806b7b0 0x7fbf88072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 -- 172.21.15.116:0/866859394 >> v2:172.21.15.129:3300/0 conn(0x7fbf8806b7b0 msgr2=0x7fbf88072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 --2- 172.21.15.116:0/866859394 >> v2:172.21.15.129:3300/0 conn(0x7fbf8806b7b0 0x7fbf88072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 -- 172.21.15.116:0/866859394 >> v1:172.21.15.129:6789/0 conn(0x7fbf880b0880 legacy=0x7fbf880aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 -- 172.21.15.116:0/866859394 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf880b43f0 con 0x7fbf880ad1f0 2023-11-08T08:19:45.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8d7da700 1 --2- 172.21.15.116:0/866859394 >> v2:172.21.15.129:3300/0 conn(0x7fbf8806b7b0 0x7fbf88072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:45.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8e7dc700 1 --2- 172.21.15.116:0/866859394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf880ad1f0 0x7fbf880ad610 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbf8806f560 tx=0x7fbf8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c309d3b3682f234a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8cfd9700 1 -- 172.21.15.116:0/866859394 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000e070 con 0x7fbf880ad1f0 2023-11-08T08:19:45.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8cfd9700 1 -- 172.21.15.116:0/866859394 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf80003a20 con 0x7fbf880ad1f0 2023-11-08T08:19:45.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.556+0000 7fbf8cfd9700 1 -- 172.21.15.116:0/866859394 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80003dd0 con 0x7fbf880ad1f0 2023-11-08T08:19:45.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 -- 172.21.15.116:0/866859394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf880ad1f0 msgr2=0x7fbf880ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 --2- 172.21.15.116:0/866859394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf880ad1f0 0x7fbf880ad610 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbf8806f560 tx=0x7fbf8000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:45.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 -- 172.21.15.116:0/866859394 shutdown_connections 2023-11-08T08:19:45.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 --2- 172.21.15.116:0/866859394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf880ad1f0 0x7fbf880ad610 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 --2- 172.21.15.116:0/866859394 >> v2:172.21.15.129:3300/0 conn(0x7fbf8806b7b0 0x7fbf88072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 -- 172.21.15.116:0/866859394 >> 172.21.15.116:0/866859394 conn(0x7fbf880b2fc0 msgr2=0x7fbf88071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:45.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 -- 172.21.15.116:0/866859394 shutdown_connections 2023-11-08T08:19:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.557+0000 7fbf9023f700 1 -- 172.21.15.116:0/866859394 wait complete. 2023-11-08T08:19:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.559+0000 7fbf9023f700 1 Processor -- start 2023-11-08T08:19:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.559+0000 7fbf9023f700 1 -- start start 2023-11-08T08:19:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 0x7fbf8815c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf880ad1f0 0x7fbf880cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf880b0880 0x7fbf881f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881f2030 con 0x7fbf8806b7b0 2023-11-08T08:19:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8809b490 con 0x7fbf880ad1f0 2023-11-08T08:19:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8809b770 con 0x7fbf880b0880 2023-11-08T08:19:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 0x7fbf8815c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 0x7fbf8815c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60148/0 (socket says 172.21.15.116:60148) 2023-11-08T08:19:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 -- 172.21.15.116:0/836450712 learned_addr learned my addr 172.21.15.116:0/836450712 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8d7da700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf880ad1f0 0x7fbf880cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8e7dc700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf880b0880 0x7fbf881f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 -- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf880b0880 msgr2=0x7fbf881f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf880b0880 0x7fbf881f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 -- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf880ad1f0 msgr2=0x7fbf880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf880ad1f0 0x7fbf880cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 -- 172.21.15.116:0/836450712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80009b30 con 0x7fbf8806b7b0 2023-11-08T08:19:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8e7dc700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf880b0880 0x7fbf881f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf8dfdb700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 0x7fbf8815c4d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400c4d0 tx=0x7fbf8400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84011790 con 0x7fbf8806b7b0 2023-11-08T08:19:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf840118f0 con 0x7fbf8806b7b0 2023-11-08T08:19:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf880b4d00 con 0x7fbf8806b7b0 2023-11-08T08:19:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84011ca0 con 0x7fbf8806b7b0 2023-11-08T08:19:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.560+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf880b46d0 con 0x7fbf8806b7b0 2023-11-08T08:19:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.561+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf880b4fe0 con 0x7fbf8806b7b0 2023-11-08T08:19:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.562+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf84010050 con 0x7fbf8806b7b0 2023-11-08T08:19:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.562+0000 7fbf7e7fc700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbf6c0682f0 0x7fbf6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.562+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbf840858e0 con 0x7fbf8806b7b0 2023-11-08T08:19:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.562+0000 7fbf8d7da700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbf6c0682f0 0x7fbf6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.562+0000 7fbf8d7da700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbf6c0682f0 0x7fbf6c06a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbf8806c6f0 tx=0x7fbf78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.566+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf84052140 con 0x7fbf8806b7b0 2023-11-08T08:19:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.748+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf880743f0 con 0x7fbf8806b7b0 2023-11-08T08:19:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.749+0000 7fbf7e7fc700 1 -- 172.21.15.116:0/836450712 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+17982 (secure 0 0 0) 0x7fbf84055760 con 0x7fbf8806b7b0 2023-11-08T08:19:45.749 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:45.064246+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbf6c0682f0 msgr2=0x7fbf6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbf6c0682f0 0x7fbf6c06a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbf8806c6f0 tx=0x7fbf78008040 comp rx=0 tx=0).stop 2023-11-08T08:19:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 msgr2=0x7fbf8815c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 0x7fbf8815c4d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400c4d0 tx=0x7fbf8400c7e0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 shutdown_connections 2023-11-08T08:19:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbf880b0880 0x7fbf881f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf8806b7b0 0x7fbf8815c4d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbf6c0682f0 0x7fbf6c06a7a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 --2- 172.21.15.116:0/836450712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf880ad1f0 0x7fbf880cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 >> 172.21.15.116:0/836450712 conn(0x7fbf880b2fc0 msgr2=0x7fbf8806dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 shutdown_connections 2023-11-08T08:19:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.751+0000 7fbf9023f700 1 -- 172.21.15.116:0/836450712 wait complete. 2023-11-08T08:19:45.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-08T08:19:45.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.886+0000 7f17b10c3700 1 Processor -- start 2023-11-08T08:19:45.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.886+0000 7f17b10c3700 1 -- start start 2023-11-08T08:19:45.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17b10c3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 0x7f17ac15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17b10c3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac159ec0 con 0x7f17ac0b1650 2023-11-08T08:19:45.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17b10c3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac15d8a0 con 0x7f17ac1599b0 2023-11-08T08:19:45.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17b10c3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac15db80 con 0x7f17ac0af0a0 2023-11-08T08:19:45.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17aad9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17ac0b1650 0x7f17ac0b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36518/0 (socket says 172.21.15.116:36518) 2023-11-08T08:19:45.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17aad9d700 1 -- 172.21.15.116:0/2050905075 learned_addr learned my addr 172.21.15.116:0/2050905075 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:45.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17ab59e700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 0x7f17ac15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547374687 0 0) 0x7f17ac159ec0 con 0x7f17ac0b1650 2023-11-08T08:19:45.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.887+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17940035b0 con 0x7f17ac0b1650 2023-11-08T08:19:45.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072740605 0 0) 0x7f17940035b0 con 0x7f17ac0b1650 2023-11-08T08:19:45.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 msgr2=0x7f17ac15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 0x7f17ac15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 >> v1:172.21.15.129:6789/0 conn(0x7f17ac1599b0 legacy=0x7f17ac15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac15de60 con 0x7f17ac0b1650 2023-11-08T08:19:45.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f17a0001670 con 0x7f17ac0b1650 2023-11-08T08:19:45.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 >> v1:172.21.15.116:6789/0 conn(0x7f17ac0b1650 legacy=0x7f17ac0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 0x7f1794003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1794003fb0 0x7f1794004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1794004d90 0x7f1794005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac15db80 con 0x7f1794003fb0 2023-11-08T08:19:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179400baf0 con 0x7f1794004d90 2023-11-08T08:19:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179400bcb0 con 0x7f17ac0af0a0 2023-11-08T08:19:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1794003fb0 0x7f1794004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aad9d700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 0x7f1794003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17ab59e700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1794004d90 0x7f1794005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 -- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 msgr2=0x7f1794003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17ac0af0a0 0x7f1794003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 -- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1794004d90 msgr2=0x7f1794005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1794004d90 0x7f1794005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 -- 172.21.15.116:0/2050905075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17940035b0 con 0x7f1794003fb0 2023-11-08T08:19:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17ab59e700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1794004d90 0x7f1794005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17aa59c700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1794003fb0 0x7f1794004850 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f179c007fd0 tx=0x7f1794005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9d76347903c6ed server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0012f0 con 0x7f1794003fb0 2023-11-08T08:19:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c005220 con 0x7f1794003fb0 2023-11-08T08:19:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.888+0000 7f17a9d9b700 1 -- 172.21.15.116:0/2050905075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c006860 con 0x7f1794003fb0 2023-11-08T08:19:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17b10c3700 1 -- 172.21.15.116:0/2050905075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1794003fb0 msgr2=0x7f1794004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17b10c3700 1 --2- 172.21.15.116:0/2050905075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1794003fb0 0x7f1794004850 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f179c007fd0 tx=0x7f1794005880 comp rx=0 tx=0).stop 2023-11-08T08:19:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17ab59e700 1 -- 172.21.15.116:0/2050905075 reap_dead start 2023-11-08T08:19:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17b10c3700 1 -- 172.21.15.116:0/2050905075 shutdown_connections 2023-11-08T08:19:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17b10c3700 1 -- 172.21.15.116:0/2050905075 >> 172.21.15.116:0/2050905075 conn(0x7f17ac06c010 msgr2=0x7f17ac0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17b10c3700 1 -- 172.21.15.116:0/2050905075 shutdown_connections 2023-11-08T08:19:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.889+0000 7f17b10c3700 1 -- 172.21.15.116:0/2050905075 wait complete. 2023-11-08T08:19:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.890+0000 7f17b10c3700 1 Processor -- start 2023-11-08T08:19:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 -- start start 2023-11-08T08:19:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1794003fb0 0x7f17ac152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 0x7f17ac1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17ac0b56d0 0x7f17ac1e5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac156e00 con 0x7f17ac0b5260 2023-11-08T08:19:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac0b3f20 con 0x7f17ac0b56d0 2023-11-08T08:19:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17b10c3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac0b41c0 con 0x7f1794003fb0 2023-11-08T08:19:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17aa59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 0x7f17ac1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17aa59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 0x7f17ac1533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60168/0 (socket says 172.21.15.116:60168) 2023-11-08T08:19:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.891+0000 7f17aa59c700 1 -- 172.21.15.116:0/3597347024 learned_addr learned my addr 172.21.15.116:0/3597347024 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17ab59e700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17ac0b56d0 0x7f17ac1e5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aad9d700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1794003fb0 0x7f17ac152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aa59c700 1 -- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1794003fb0 msgr2=0x7f17ac152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aa59c700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1794003fb0 0x7f17ac152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aa59c700 1 -- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17ac0b56d0 msgr2=0x7f17ac1e5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aa59c700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17ac0b56d0 0x7f17ac1e5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aa59c700 1 -- 172.21.15.116:0/3597347024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac15a4e0 con 0x7f17ac0b5260 2023-11-08T08:19:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17ab59e700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17ac0b56d0 0x7f17ac1e5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17aa59c700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 0x7f17ac1533d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f17ac06eb00 tx=0x7f179c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c006e70 con 0x7f17ac0b5260 2023-11-08T08:19:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c001290 con 0x7f17ac0b5260 2023-11-08T08:19:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac15a350 con 0x7f17ac0b5260 2023-11-08T08:19:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0074a0 con 0x7f17ac0b5260 2023-11-08T08:19:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.892+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac15a670 con 0x7f17ac0b5260 2023-11-08T08:19:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.893+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac071be0 con 0x7f17ac0b5260 2023-11-08T08:19:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.893+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f179c010050 con 0x7f17ac0b5260 2023-11-08T08:19:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.894+0000 7f17937fe700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17980682f0 0x7f179806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.894+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f179c086db0 con 0x7f17ac0b5260 2023-11-08T08:19:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.894+0000 7f17aad9d700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17980682f0 0x7f179806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.894+0000 7f17aad9d700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17980682f0 0x7f179806a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f17a0001050 tx=0x7f17a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:45.898+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f179c053610 con 0x7f17ac0b5260 2023-11-08T08:19:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:46.119+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f17ac0743f0 con 0x7f17ac0b5260 2023-11-08T08:19:47.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.073+0000 7f17937fe700 1 -- 172.21.15.116:0/3597347024 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f179c056c30 con 0x7f17ac0b5260 2023-11-08T08:19:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17980682f0 msgr2=0x7f179806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17980682f0 0x7f179806a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f17a0001050 tx=0x7f17a0009040 comp rx=0 tx=0).stop 2023-11-08T08:19:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 msgr2=0x7f17ac1533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 0x7f17ac1533d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f17ac06eb00 tx=0x7f179c001120 comp rx=0 tx=0).stop 2023-11-08T08:19:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 shutdown_connections 2023-11-08T08:19:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17980682f0 0x7f179806a7a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17ac0b56d0 0x7f17ac1e5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1794003fb0 0x7f17ac152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 --2- 172.21.15.116:0/3597347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17ac0b5260 0x7f17ac1533d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 >> 172.21.15.116:0/3597347024 conn(0x7f17ac06c010 msgr2=0x7f17ac15b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:47.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 shutdown_connections 2023-11-08T08:19:47.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.076+0000 7f17b10c3700 1 -- 172.21.15.116:0/3597347024 wait complete. 2023-11-08T08:19:47.090 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-08T08:19:47.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-08T08:19:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.219+0000 7fc12cc26700 1 Processor -- start 2023-11-08T08:19:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.219+0000 7fc12cc26700 1 -- start start 2023-11-08T08:19:47.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12cc26700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1280cf710 0x7fc1280ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:47.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12cc26700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc128071890 0x7fc128071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:47.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12cc26700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12815cc10 con 0x7fc12806a270 2023-11-08T08:19:47.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12cc26700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12806c9b0 con 0x7fc1280cf710 2023-11-08T08:19:47.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12cc26700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12806cc90 con 0x7fc128071890 2023-11-08T08:19:47.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1280cf710 0x7fc1280ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:47.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc126d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc12806a270 0x7fc1280b1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36520/0 (socket says 172.21.15.116:36520) 2023-11-08T08:19:47.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1280cf710 0x7fc1280ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60180/0 (socket says 172.21.15.116:60180) 2023-11-08T08:19:47.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc126d9d700 1 -- 172.21.15.116:0/3140758136 learned_addr learned my addr 172.21.15.116:0/3140758136 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:47.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc125d9b700 1 --2- 172.21.15.116:0/3140758136 >> v2:172.21.15.129:3300/0 conn(0x7fc128071890 0x7fc128071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:47.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 -- 172.21.15.116:0/3140758136 >> v2:172.21.15.129:3300/0 conn(0x7fc128071890 msgr2=0x7fc128071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:47.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 --2- 172.21.15.116:0/3140758136 >> v2:172.21.15.129:3300/0 conn(0x7fc128071890 0x7fc128071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 -- 172.21.15.116:0/3140758136 >> v1:172.21.15.116:6789/0 conn(0x7fc12806a270 legacy=0x7fc1280b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:47.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 -- 172.21.15.116:0/3140758136 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12806cf70 con 0x7fc1280cf710 2023-11-08T08:19:47.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12559a700 1 -- 172.21.15.116:0/3140758136 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797965138 0 0) 0x7fc12815cc10 con 0x7fc12806a270 2023-11-08T08:19:47.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12659c700 1 --2- 172.21.15.116:0/3140758136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1280cf710 0x7fc1280ce740 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc118009840 tx=0x7fc118009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a08088ae2b35e99 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:47.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12559a700 1 -- 172.21.15.116:0/3140758136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118011830 con 0x7fc1280cf710 2023-11-08T08:19:47.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12559a700 1 -- 172.21.15.116:0/3140758136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc118011990 con 0x7fc1280cf710 2023-11-08T08:19:47.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.220+0000 7fc12559a700 1 -- 172.21.15.116:0/3140758136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118011d20 con 0x7fc1280cf710 2023-11-08T08:19:47.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 -- 172.21.15.116:0/3140758136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1280cf710 msgr2=0x7fc1280ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:47.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 --2- 172.21.15.116:0/3140758136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1280cf710 0x7fc1280ce740 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc118009840 tx=0x7fc118009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 -- 172.21.15.116:0/3140758136 shutdown_connections 2023-11-08T08:19:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 --2- 172.21.15.116:0/3140758136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1280cf710 0x7fc1280ce740 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 --2- 172.21.15.116:0/3140758136 >> v2:172.21.15.129:3300/0 conn(0x7fc128071890 0x7fc128071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 -- 172.21.15.116:0/3140758136 >> 172.21.15.116:0/3140758136 conn(0x7fc12806c010 msgr2=0x7fc1280cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 -- 172.21.15.116:0/3140758136 shutdown_connections 2023-11-08T08:19:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.221+0000 7fc12cc26700 1 -- 172.21.15.116:0/3140758136 wait complete. 2023-11-08T08:19:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.222+0000 7fc12cc26700 1 Processor -- start 2023-11-08T08:19:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.222+0000 7fc12cc26700 1 -- start start 2023-11-08T08:19:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc12cc26700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc12806a270 0x7fc1280aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc12cc26700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 0x7fc128143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc12cc26700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1280cf710 0x7fc1281470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc12cc26700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280af520 con 0x7fc128071890 2023-11-08T08:19:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc12cc26700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815fce0 con 0x7fc1280cf710 2023-11-08T08:19:47.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc12cc26700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815d270 con 0x7fc12806a270 2023-11-08T08:19:47.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 0x7fc128143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:47.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 0x7fc128143ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60182/0 (socket says 172.21.15.116:60182) 2023-11-08T08:19:47.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 -- 172.21.15.116:0/2146871040 learned_addr learned my addr 172.21.15.116:0/2146871040 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:47.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 -- 172.21.15.116:0/2146871040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc12806a270 msgr2=0x7fc1280aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc12806a270 0x7fc1280aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 -- 172.21.15.116:0/2146871040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1280cf710 msgr2=0x7fc1281470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1280cf710 0x7fc1281470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.223+0000 7fc125d9b700 1 -- 172.21.15.116:0/2146871040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180094a0 con 0x7fc128071890 2023-11-08T08:19:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.224+0000 7fc125d9b700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 0x7fc128143ea0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00e810 tx=0x7fc11c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.224+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c013610 con 0x7fc128071890 2023-11-08T08:19:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.224+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c013770 con 0x7fc128071890 2023-11-08T08:19:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.224+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c013b20 con 0x7fc128071890 2023-11-08T08:19:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.224+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12806d530 con 0x7fc128071890 2023-11-08T08:19:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.224+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12806d250 con 0x7fc128071890 2023-11-08T08:19:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.225+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12806a6b0 con 0x7fc128071890 2023-11-08T08:19:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.225+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc11c023070 con 0x7fc128071890 2023-11-08T08:19:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.226+0000 7fc116ffd700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1080682f0 0x7fc10806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.226+0000 7fc12659c700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1080682f0 0x7fc10806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.226+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc11c085970 con 0x7fc128071890 2023-11-08T08:19:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.226+0000 7fc12659c700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1080682f0 0x7fc10806a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.230+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc11c052150 con 0x7fc128071890 2023-11-08T08:19:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:47.453+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc1280743f0 con 0x7fc128071890 2023-11-08T08:19:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.078+0000 7fc116ffd700 1 -- 172.21.15.116:0/2146871040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc11c055770 con 0x7fc128071890 2023-11-08T08:19:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1080682f0 msgr2=0x7fc10806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1080682f0 0x7fc10806a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118010000 comp rx=0 tx=0).stop 2023-11-08T08:19:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 msgr2=0x7fc128143ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 0x7fc128143ea0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00e810 tx=0x7fc11c00ebd0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 shutdown_connections 2023-11-08T08:19:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1080682f0 0x7fc10806a7a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1280cf710 0x7fc1281470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc12806a270 0x7fc1280aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 --2- 172.21.15.116:0/2146871040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc128071890 0x7fc128143ea0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 >> 172.21.15.116:0/2146871040 conn(0x7fc12806c010 msgr2=0x7fc1280b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 shutdown_connections 2023-11-08T08:19:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.080+0000 7fc12cc26700 1 -- 172.21.15.116:0/2146871040 wait complete. 2023-11-08T08:19:48.094 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-08T08:19:48.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:19:48.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.217+0000 7f989627c700 1 Processor -- start 2023-11-08T08:19:48.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.217+0000 7f989627c700 1 -- start start 2023-11-08T08:19:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f989627c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98900b22a0 0x7f98900cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f989627c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900d1ee0 con 0x7f989006ab10 2023-11-08T08:19:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f989627c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900cdb10 con 0x7f98900cd5a0 2023-11-08T08:19:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f989627c700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98900d1890 con 0x7f98900b22a0 2023-11-08T08:19:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98900b22a0 0x7f98900cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98900b22a0 0x7f98900cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60186/0 (socket says 172.21.15.116:60186) 2023-11-08T08:19:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f988ffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f989006ab10 0x7f989006aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36536/0 (socket says 172.21.15.116:36536) 2023-11-08T08:19:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 -- 172.21.15.116:0/1718708915 learned_addr learned my addr 172.21.15.116:0/1718708915 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 -- 172.21.15.116:0/1718708915 >> v1:172.21.15.129:6789/0 conn(0x7f98900cd5a0 legacy=0x7f98900cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 -- 172.21.15.116:0/1718708915 >> v1:172.21.15.116:6789/0 conn(0x7f989006ab10 legacy=0x7f989006aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 -- 172.21.15.116:0/1718708915 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900d1b70 con 0x7f98900b22a0 2023-11-08T08:19:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f988effd700 1 -- 172.21.15.116:0/1718708915 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962391945 0 0) 0x7f98900d1ee0 con 0x7f989006ab10 2023-11-08T08:19:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f9894819700 1 --2- 172.21.15.116:0/1718708915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98900cfb50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9888009e50 tx=0x7f988800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994eddf6ac63f733 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f988effd700 1 -- 172.21.15.116:0/1718708915 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988800e070 con 0x7f98900b22a0 2023-11-08T08:19:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.218+0000 7f988effd700 1 -- 172.21.15.116:0/1718708915 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9888003a20 con 0x7f98900b22a0 2023-11-08T08:19:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f988effd700 1 -- 172.21.15.116:0/1718708915 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888003dd0 con 0x7f98900b22a0 2023-11-08T08:19:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 -- 172.21.15.116:0/1718708915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 msgr2=0x7f98900cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 --2- 172.21.15.116:0/1718708915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98900cfb50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9888009e50 tx=0x7f988800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 -- 172.21.15.116:0/1718708915 shutdown_connections 2023-11-08T08:19:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 --2- 172.21.15.116:0/1718708915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98900cfb50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 -- 172.21.15.116:0/1718708915 >> 172.21.15.116:0/1718708915 conn(0x7f989006c520 msgr2=0x7f989006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 -- 172.21.15.116:0/1718708915 shutdown_connections 2023-11-08T08:19:48.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.219+0000 7f989627c700 1 -- 172.21.15.116:0/1718708915 wait complete. 2023-11-08T08:19:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.220+0000 7f989627c700 1 Processor -- start 2023-11-08T08:19:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.220+0000 7f989627c700 1 -- start start 2023-11-08T08:19:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f989627c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989006ab10 0x7f9890157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f989627c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98901f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f989627c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98900cd5a0 0x7f98900b8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f989627c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9890157a70 con 0x7f98900b22a0 2023-11-08T08:19:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f989627c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9890158e20 con 0x7f989006ab10 2023-11-08T08:19:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f989627c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98901590c0 con 0x7f98900cd5a0 2023-11-08T08:19:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f988f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98901f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f988f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98901f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60190/0 (socket says 172.21.15.116:60190) 2023-11-08T08:19:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f988f7fe700 1 -- 172.21.15.116:0/1979975111 learned_addr learned my addr 172.21.15.116:0/1979975111 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f988ffff700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989006ab10 0x7f9890157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.221+0000 7f9894819700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98900cd5a0 0x7f98900b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988f7fe700 1 -- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98900cd5a0 msgr2=0x7f98900b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988f7fe700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98900cd5a0 0x7f98900b8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988f7fe700 1 -- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989006ab10 msgr2=0x7f9890157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988f7fe700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989006ab10 0x7f9890157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988f7fe700 1 -- 172.21.15.116:0/1979975111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888009b30 con 0x7f98900b22a0 2023-11-08T08:19:48.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f9894819700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98900cd5a0 0x7f98900b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:48.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988f7fe700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98901f26d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f988000b450 tx=0x7f988000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880011830 con 0x7f98900b22a0 2023-11-08T08:19:48.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9880011990 con 0x7f98900b22a0 2023-11-08T08:19:48.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98900b1ec0 con 0x7f98900b22a0 2023-11-08T08:19:48.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880011d40 con 0x7f98900b22a0 2023-11-08T08:19:48.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.222+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989015cad0 con 0x7f98900b22a0 2023-11-08T08:19:48.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.223+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98900b2050 con 0x7f98900b22a0 2023-11-08T08:19:48.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.223+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f988000f050 con 0x7f98900b22a0 2023-11-08T08:19:48.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.223+0000 7f988cff9700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f987c0682f0 0x7f987c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.224+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9880085810 con 0x7f98900b22a0 2023-11-08T08:19:48.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.224+0000 7f988ffff700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f987c0682f0 0x7f987c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.224+0000 7f988ffff700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f987c0682f0 0x7f987c06a7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9884007fd0 tx=0x7f9884007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.228+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9880051f20 con 0x7f98900b22a0 2023-11-08T08:19:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.448+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98900743f0 con 0x7f98900b22a0 2023-11-08T08:19:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.448+0000 7f988cff9700 1 -- 172.21.15.116:0/1979975111 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f9880055540 con 0x7f98900b22a0 2023-11-08T08:19:48.449 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:48.449 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:19:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f987c0682f0 msgr2=0x7f987c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f987c0682f0 0x7f987c06a7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9884007fd0 tx=0x7f9884007de0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 msgr2=0x7f98901f26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98901f26d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f988000b450 tx=0x7f988000b760 comp rx=0 tx=0).stop 2023-11-08T08:19:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 shutdown_connections 2023-11-08T08:19:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98900cd5a0 0x7f98900b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f987c0682f0 0x7f987c06a7a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989006ab10 0x7f9890157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 --2- 172.21.15.116:0/1979975111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98900b22a0 0x7f98901f26d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 >> 172.21.15.116:0/1979975111 conn(0x7f989006c520 msgr2=0x7f98900b65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 shutdown_connections 2023-11-08T08:19:48.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.450+0000 7f989627c700 1 -- 172.21.15.116:0/1979975111 wait complete. 2023-11-08T08:19:48.453 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:19:48.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:48.576 INFO:tasks.ceph.mds.a.smithi116.stderr:2023-11-08T08:19:48.576+0000 7fe0a79d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:19:48.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.582+0000 7efdb782f700 1 Processor -- start 2023-11-08T08:19:48.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.582+0000 7efdb782f700 1 -- start start 2023-11-08T08:19:48.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb782f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efdb015cc80 0x7efdb006de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb782f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdb006e3a0 0x7efdb0070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb782f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb00b1810 con 0x7efdb006a270 2023-11-08T08:19:48.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb782f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00b2490 con 0x7efdb006e3a0 2023-11-08T08:19:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb782f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00b1af0 con 0x7efdb015cc80 2023-11-08T08:19:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb5dcc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdb006e3a0 0x7efdb0070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb55cb700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efdb006a270 0x7efdb006a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36540/0 (socket says 172.21.15.116:36540) 2023-11-08T08:19:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb55cb700 1 -- 172.21.15.116:0/4045527599 learned_addr learned my addr 172.21.15.116:0/4045527599 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb5dcc700 1 -- 172.21.15.116:0/4045527599 >> v2:172.21.15.129:3300/0 conn(0x7efdb015cc80 msgr2=0x7efdb006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb5dcc700 1 --2- 172.21.15.116:0/4045527599 >> v2:172.21.15.129:3300/0 conn(0x7efdb015cc80 0x7efdb006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb5dcc700 1 -- 172.21.15.116:0/4045527599 >> v1:172.21.15.116:6789/0 conn(0x7efdb006a270 legacy=0x7efdb006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb5dcc700 1 -- 172.21.15.116:0/4045527599 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00b1dd0 con 0x7efdb006e3a0 2023-11-08T08:19:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efda7fff700 1 -- 172.21.15.116:0/4045527599 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918012256 0 0) 0x7efdb00b1810 con 0x7efdb006a270 2023-11-08T08:19:48.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efdb5dcc700 1 --2- 172.21.15.116:0/4045527599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006e3a0 0x7efdb0070d20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7efdb006c6c0 tx=0x7efda800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a7fe10bb9433d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efda7fff700 1 -- 172.21.15.116:0/4045527599 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda800e070 con 0x7efdb006e3a0 2023-11-08T08:19:48.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efda7fff700 1 -- 172.21.15.116:0/4045527599 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda800bbd0 con 0x7efdb006e3a0 2023-11-08T08:19:48.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.583+0000 7efda7fff700 1 -- 172.21.15.116:0/4045527599 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8012430 con 0x7efdb006e3a0 2023-11-08T08:19:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 -- 172.21.15.116:0/4045527599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006e3a0 msgr2=0x7efdb0070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 --2- 172.21.15.116:0/4045527599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006e3a0 0x7efdb0070d20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7efdb006c6c0 tx=0x7efda800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 -- 172.21.15.116:0/4045527599 shutdown_connections 2023-11-08T08:19:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 --2- 172.21.15.116:0/4045527599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006e3a0 0x7efdb0070d20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 --2- 172.21.15.116:0/4045527599 >> v2:172.21.15.129:3300/0 conn(0x7efdb015cc80 0x7efdb006de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 -- 172.21.15.116:0/4045527599 >> 172.21.15.116:0/4045527599 conn(0x7efdb006c010 msgr2=0x7efdb00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 -- 172.21.15.116:0/4045527599 shutdown_connections 2023-11-08T08:19:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.584+0000 7efdb782f700 1 -- 172.21.15.116:0/4045527599 wait complete. 2023-11-08T08:19:48.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.585+0000 7efdb782f700 1 Processor -- start 2023-11-08T08:19:48.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.585+0000 7efdb782f700 1 -- start start 2023-11-08T08:19:48.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb782f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 0x7efdb00b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb782f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006e3a0 0x7efdb00c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb782f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb015cc80 0x7efdb00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb782f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c4d80 con 0x7efdb006a270 2023-11-08T08:19:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb782f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c5060 con 0x7efdb006e3a0 2023-11-08T08:19:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb782f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c5340 con 0x7efdb015cc80 2023-11-08T08:19:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 0x7efdb00b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 0x7efdb00b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60214/0 (socket says 172.21.15.116:60214) 2023-11-08T08:19:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 -- 172.21.15.116:0/3488540416 learned_addr learned my addr 172.21.15.116:0/3488540416 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 -- 172.21.15.116:0/3488540416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb015cc80 msgr2=0x7efdb00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb015cc80 0x7efdb00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 -- 172.21.15.116:0/3488540416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006e3a0 msgr2=0x7efdb00c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006e3a0 0x7efdb00c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.586+0000 7efdb55cb700 1 -- 172.21.15.116:0/3488540416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda8009a70 con 0x7efdb006a270 2023-11-08T08:19:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.587+0000 7efdb55cb700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 0x7efdb00b5460 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7efdac00c550 tx=0x7efdac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.587+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac004c80 con 0x7efdb006a270 2023-11-08T08:19:48.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.587+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac004e00 con 0x7efdb006a270 2023-11-08T08:19:48.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.587+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb006e7c0 con 0x7efdb006a270 2023-11-08T08:19:48.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.587+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac007310 con 0x7efdb006a270 2023-11-08T08:19:48.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.587+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb00b20b0 con 0x7efdb006a270 2023-11-08T08:19:48.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.588+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb006e950 con 0x7efdb006a270 2023-11-08T08:19:48.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.588+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efdac01a020 con 0x7efdb006a270 2023-11-08T08:19:48.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.588+0000 7efda5ffb700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd940682f0 0x7efd9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.589+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efdac085850 con 0x7efdb006a270 2023-11-08T08:19:48.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.589+0000 7efdb4dca700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd940682f0 0x7efd9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.589+0000 7efdb4dca700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd940682f0 0x7efd9406a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efdb00b2bf0 tx=0x7efda0006d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.593+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efdb006e950 con 0x7efdb006a270 2023-11-08T08:19:48.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.774+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdb006e950 con 0x7efdb006a270 2023-11-08T08:19:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.775+0000 7efda5ffb700 1 -- 172.21.15.116:0/3488540416 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+17969 (secure 0 0 0) 0x7efdb006e950 con 0x7efdb006a270 2023-11-08T08:19:48.775 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:48.776 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:46.066442+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:48.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd940682f0 msgr2=0x7efd9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd940682f0 0x7efd9406a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efdb00b2bf0 tx=0x7efda0006d20 comp rx=0 tx=0).stop 2023-11-08T08:19:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 msgr2=0x7efdb00b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 0x7efdb00b5460 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7efdac00c550 tx=0x7efdac00c910 comp rx=0 tx=0).stop 2023-11-08T08:19:48.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 shutdown_connections 2023-11-08T08:19:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7efdb015cc80 0x7efdb00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb006a270 0x7efdb00b5460 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efd940682f0 0x7efd9406a7a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 --2- 172.21.15.116:0/3488540416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006e3a0 0x7efdb00c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 >> 172.21.15.116:0/3488540416 conn(0x7efdb006c010 msgr2=0x7efdb006a640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:48.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 shutdown_connections 2023-11-08T08:19:48.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.777+0000 7efdb782f700 1 -- 172.21.15.116:0/3488540416 wait complete. 2023-11-08T08:19:48.789 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-08T08:19:48.790 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:19:48.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:19:48.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:19:48.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.913+0000 7fad64cbc700 1 Processor -- start 2023-11-08T08:19:48.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.914+0000 7fad64cbc700 1 -- start start 2023-11-08T08:19:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.914+0000 7fad64cbc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad6006e570 0x7fad60160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.914+0000 7fad64cbc700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600b1810 con 0x7fad6015cc80 2023-11-08T08:19:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.914+0000 7fad64cbc700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600b1af0 con 0x7fad6006a270 2023-11-08T08:19:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.914+0000 7fad64cbc700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad600b2490 con 0x7fad6006e570 2023-11-08T08:19:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5dd9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fad6015cc80 0x7fad6006de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36546/0 (socket says 172.21.15.116:36546) 2023-11-08T08:19:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5dd9b700 1 -- 172.21.15.116:0/2870746195 learned_addr learned my addr 172.21.15.116:0/2870746195 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5ed9d700 1 --2- 172.21.15.116:0/2870746195 >> v2:172.21.15.116:3300/0 conn(0x7fad6006e570 0x7fad60160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5d59a700 1 -- 172.21.15.116:0/2870746195 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596617399 0 0) 0x7fad600b1810 con 0x7fad6015cc80 2023-11-08T08:19:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5d59a700 1 -- 172.21.15.116:0/2870746195 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c0035b0 con 0x7fad6015cc80 2023-11-08T08:19:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5ed9d700 1 -- 172.21.15.116:0/2870746195 >> v1:172.21.15.129:6789/0 conn(0x7fad6006a270 legacy=0x7fad6006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5ed9d700 1 -- 172.21.15.116:0/2870746195 >> v1:172.21.15.116:6789/0 conn(0x7fad6015cc80 legacy=0x7fad6006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5ed9d700 1 -- 172.21.15.116:0/2870746195 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad600b1dd0 con 0x7fad6006e570 2023-11-08T08:19:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5ed9d700 1 --2- 172.21.15.116:0/2870746195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6006e570 0x7fad60160300 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fad50009d90 tx=0x7fad5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69527c2d2764a049 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5d59a700 1 -- 172.21.15.116:0/2870746195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5000e070 con 0x7fad6006e570 2023-11-08T08:19:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5d59a700 1 -- 172.21.15.116:0/2870746195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5000bb20 con 0x7fad6006e570 2023-11-08T08:19:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.915+0000 7fad5d59a700 1 -- 172.21.15.116:0/2870746195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50012430 con 0x7fad6006e570 2023-11-08T08:19:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 -- 172.21.15.116:0/2870746195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6006e570 msgr2=0x7fad60160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 --2- 172.21.15.116:0/2870746195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6006e570 0x7fad60160300 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fad50009d90 tx=0x7fad5000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 -- 172.21.15.116:0/2870746195 shutdown_connections 2023-11-08T08:19:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 --2- 172.21.15.116:0/2870746195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6006e570 0x7fad60160300 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 -- 172.21.15.116:0/2870746195 >> 172.21.15.116:0/2870746195 conn(0x7fad6006c010 msgr2=0x7fad600cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 -- 172.21.15.116:0/2870746195 shutdown_connections 2023-11-08T08:19:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.916+0000 7fad64cbc700 1 -- 172.21.15.116:0/2870746195 wait complete. 2023-11-08T08:19:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.917+0000 7fad64cbc700 1 Processor -- start 2023-11-08T08:19:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.917+0000 7fad64cbc700 1 -- start start 2023-11-08T08:19:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad64cbc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad6006a270 0x7fad6014db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad64cbc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad6006e570 0x7fad6015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad64cbc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 0x7fad6015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad64cbc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad600c5d70 con 0x7fad6015cc80 2023-11-08T08:19:48.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad64cbc700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad600c36f0 con 0x7fad6006e570 2023-11-08T08:19:48.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad64cbc700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad600c39d0 con 0x7fad6006a270 2023-11-08T08:19:48.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad5ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 0x7fad6015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad5ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 0x7fad6015c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60230/0 (socket says 172.21.15.116:60230) 2023-11-08T08:19:48.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad5ed9d700 1 -- 172.21.15.116:0/1268560257 learned_addr learned my addr 172.21.15.116:0/1268560257 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:48.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad5dd9b700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad6006e570 0x7fad6015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.918+0000 7fad5e59c700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad6006a270 0x7fad6014db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad5ed9d700 1 -- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad6006a270 msgr2=0x7fad6014db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad5ed9d700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad6006a270 0x7fad6014db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad5ed9d700 1 -- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad6006e570 msgr2=0x7fad6015c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:48.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad5ed9d700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad6006e570 0x7fad6015c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:48.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad5ed9d700 1 -- 172.21.15.116:0/1268560257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad50009a70 con 0x7fad6015cc80 2023-11-08T08:19:48.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad5ed9d700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 0x7fad6015c610 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fad60067450 tx=0x7fad50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50012430 con 0x7fad6015cc80 2023-11-08T08:19:48.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad50020070 con 0x7fad6015cc80 2023-11-08T08:19:48.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6015daa0 con 0x7fad6015cc80 2023-11-08T08:19:48.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50004280 con 0x7fad6015cc80 2023-11-08T08:19:48.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.919+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad600b20b0 con 0x7fad6015cc80 2023-11-08T08:19:48.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.920+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6015dc30 con 0x7fad6015cc80 2023-11-08T08:19:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.920+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad5002e070 con 0x7fad6015cc80 2023-11-08T08:19:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.921+0000 7fad56ffd700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad400682f0 0x7fad4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.921+0000 7fad5e59c700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad400682f0 0x7fad4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.921+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fad50022070 con 0x7fad6015cc80 2023-11-08T08:19:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.921+0000 7fad5e59c700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad400682f0 0x7fad4006a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad58007fd0 tx=0x7fad58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:48.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:48.925+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad500540f0 con 0x7fad6015cc80 2023-11-08T08:19:49.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.105+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad600c3860 con 0x7fad400682f0 2023-11-08T08:19:49.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.114+0000 7fad56ffd700 1 -- 172.21.15.116:0/1268560257 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311434 (secure 0 0 0) 0x7fad600c3860 con 0x7fad400682f0 2023-11-08T08:19:49.118 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad400682f0 msgr2=0x7fad4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad400682f0 0x7fad4006a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad58007fd0 tx=0x7fad58009040 comp rx=0 tx=0).stop 2023-11-08T08:19:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 msgr2=0x7fad6015c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 0x7fad6015c610 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fad60067450 tx=0x7fad50004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 shutdown_connections 2023-11-08T08:19:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6015cc80 0x7fad6015c610 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fad6006a270 0x7fad6014db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fad400682f0 0x7fad4006a7a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 --2- 172.21.15.116:0/1268560257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad6006e570 0x7fad6015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 >> 172.21.15.116:0/1268560257 conn(0x7fad6006c010 msgr2=0x7fad600cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 shutdown_connections 2023-11-08T08:19:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.117+0000 7fad64cbc700 1 -- 172.21.15.116:0/1268560257 wait complete. 2023-11-08T08:19:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:19:49.129 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2023-11-08T08:19:48.365903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68792,"kb_used_data":3896,"kb_used_omap":0,"kb_used_meta":64832,"kb_avail":754905928,"statfs":{"total":773094113280,"available":773023670272,"internally_reserved":0,"allocated":3989504,"data_stored":2287467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66387968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.698305"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089268+0000","last_change":"2023-11-08T08:19:45.084959+0000","last_active":"2023-11-08T08:19:45.089268+0000","last_peered":"2023-11-08T08:19:45.089268+0000","last_clean":"2023-11-08T08:19:45.089268+0000","last_became_active":"2023-11-08T08:19:45.084174+0000","last_became_peered":"2023-11-08T08:19:45.084174+0000","last_unstale":"2023-11-08T08:19:45.089268+0000","last_undegraded":"2023-11-08T08:19:45.089268+0000","last_fullsized":"2023-11-08T08:19:45.089268+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:05:13.351069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089766+0000","last_change":"2023-11-08T08:19:45.084411+0000","last_active":"2023-11-08T08:19:45.089766+0000","last_peered":"2023-11-08T08:19:45.089766+0000","last_clean":"2023-11-08T08:19:45.089766+0000","last_became_active":"2023-11-08T08:19:45.084069+0000","last_became_peered":"2023-11-08T08:19:45.084069+0000","last_unstale":"2023-11-08T08:19:45.089766+0000","last_undegraded":"2023-11-08T08:19:45.089766+0000","last_fullsized":"2023-11-08T08:19:45.089766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:19:31.649628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085060+0000","last_change":"2023-11-08T08:19:45.080611+0000","last_active":"2023-11-08T08:19:45.085060+0000","last_peered":"2023-11-08T08:19:45.085060+0000","last_clean":"2023-11-08T08:19:45.085060+0000","last_became_active":"2023-11-08T08:19:45.080146+0000","last_became_peered":"2023-11-08T08:19:45.080146+0000","last_unstale":"2023-11-08T08:19:45.085060+0000","last_undegraded":"2023-11-08T08:19:45.085060+0000","last_fullsized":"2023-11-08T08:19:45.085060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:01:43.871569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.100002+0000","last_change":"2023-11-08T08:19:45.097923+0000","last_active":"2023-11-08T08:19:45.100002+0000","last_peered":"2023-11-08T08:19:45.100002+0000","last_clean":"2023-11-08T08:19:45.100002+0000","last_became_active":"2023-11-08T08:19:45.097685+0000","last_became_peered":"2023-11-08T08:19:45.097685+0000","last_unstale":"2023-11-08T08:19:45.100002+0000","last_undegraded":"2023-11-08T08:19:45.100002+0000","last_fullsized":"2023-11-08T08:19:45.100002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:01:17.819675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.083633+0000","last_change":"2023-11-08T08:19:45.081164+0000","last_active":"2023-11-08T08:19:45.083633+0000","last_peered":"2023-11-08T08:19:45.083633+0000","last_clean":"2023-11-08T08:19:45.083633+0000","last_became_active":"2023-11-08T08:19:45.080885+0000","last_became_peered":"2023-11-08T08:19:45.080885+0000","last_unstale":"2023-11-08T08:19:45.083633+0000","last_undegraded":"2023-11-08T08:19:45.083633+0000","last_fullsized":"2023-11-08T08:19:45.083633+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:48:47.154956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093695+0000","last_change":"2023-11-08T08:19:45.092153+0000","last_active":"2023-11-08T08:19:45.093695+0000","last_peered":"2023-11-08T08:19:45.093695+0000","last_clean":"2023-11-08T08:19:45.093695+0000","last_became_active":"2023-11-08T08:19:45.091883+0000","last_became_peered":"2023-11-08T08:19:45.091883+0000","last_unstale":"2023-11-08T08:19:45.093695+0000","last_undegraded":"2023-11-08T08:19:45.093695+0000","last_fullsized":"2023-11-08T08:19:45.093695+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:35:39.702801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089888+0000","last_change":"2023-11-08T08:19:45.084859+0000","last_active":"2023-11-08T08:19:45.089888+0000","last_peered":"2023-11-08T08:19:45.089888+0000","last_clean":"2023-11-08T08:19:45.089888+0000","last_became_active":"2023-11-08T08:19:45.084551+0000","last_became_peered":"2023-11-08T08:19:45.084551+0000","last_unstale":"2023-11-08T08:19:45.089888+0000","last_undegraded":"2023-11-08T08:19:45.089888+0000","last_fullsized":"2023-11-08T08:19:45.089888+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:48:10.215458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089547+0000","last_change":"2023-11-08T08:19:45.086738+0000","last_active":"2023-11-08T08:19:45.089547+0000","last_peered":"2023-11-08T08:19:45.089547+0000","last_clean":"2023-11-08T08:19:45.089547+0000","last_became_active":"2023-11-08T08:19:45.086426+0000","last_became_peered":"2023-11-08T08:19:45.086426+0000","last_unstale":"2023-11-08T08:19:45.089547+0000","last_undegraded":"2023-11-08T08:19:45.089547+0000","last_fullsized":"2023-11-08T08:19:45.089547+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:24:01.543485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093211+0000","last_change":"2023-11-08T08:19:45.089401+0000","last_active":"2023-11-08T08:19:45.093211+0000","last_peered":"2023-11-08T08:19:45.093211+0000","last_clean":"2023-11-08T08:19:45.093211+0000","last_became_active":"2023-11-08T08:19:45.089148+0000","last_became_peered":"2023-11-08T08:19:45.089148+0000","last_unstale":"2023-11-08T08:19:45.093211+0000","last_undegraded":"2023-11-08T08:19:45.093211+0000","last_fullsized":"2023-11-08T08:19:45.093211+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:24:20.826972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097370+0000","last_change":"2023-11-08T08:19:45.095504+0000","last_active":"2023-11-08T08:19:45.097370+0000","last_peered":"2023-11-08T08:19:45.097370+0000","last_clean":"2023-11-08T08:19:45.097370+0000","last_became_active":"2023-11-08T08:19:45.095233+0000","last_became_peered":"2023-11-08T08:19:45.095233+0000","last_unstale":"2023-11-08T08:19:45.097370+0000","last_undegraded":"2023-11-08T08:19:45.097370+0000","last_fullsized":"2023-11-08T08:19:45.097370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:14:20.829614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092909+0000","last_change":"2023-11-08T08:19:45.086885+0000","last_active":"2023-11-08T08:19:45.092909+0000","last_peered":"2023-11-08T08:19:45.092909+0000","last_clean":"2023-11-08T08:19:45.092909+0000","last_became_active":"2023-11-08T08:19:45.086645+0000","last_became_peered":"2023-11-08T08:19:45.086645+0000","last_unstale":"2023-11-08T08:19:45.092909+0000","last_undegraded":"2023-11-08T08:19:45.092909+0000","last_fullsized":"2023-11-08T08:19:45.092909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:35:39.702801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087482+0000","last_change":"2023-11-08T08:19:45.085758+0000","last_active":"2023-11-08T08:19:45.087482+0000","last_peered":"2023-11-08T08:19:45.087482+0000","last_clean":"2023-11-08T08:19:45.087482+0000","last_became_active":"2023-11-08T08:19:45.085129+0000","last_became_peered":"2023-11-08T08:19:45.085129+0000","last_unstale":"2023-11-08T08:19:45.087482+0000","last_undegraded":"2023-11-08T08:19:45.087482+0000","last_fullsized":"2023-11-08T08:19:45.087482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:56.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097382+0000","last_change":"2023-11-08T08:19:45.089179+0000","last_active":"2023-11-08T08:19:45.097382+0000","last_peered":"2023-11-08T08:19:45.097382+0000","last_clean":"2023-11-08T08:19:45.097382+0000","last_became_active":"2023-11-08T08:19:45.088938+0000","last_became_peered":"2023-11-08T08:19:45.088938+0000","last_unstale":"2023-11-08T08:19:45.097382+0000","last_undegraded":"2023-11-08T08:19:45.097382+0000","last_fullsized":"2023-11-08T08:19:45.097382+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:25:03.665342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090801+0000","last_change":"2023-11-08T08:19:45.089016+0000","last_active":"2023-11-08T08:19:45.090801+0000","last_peered":"2023-11-08T08:19:45.090801+0000","last_clean":"2023-11-08T08:19:45.090801+0000","last_became_active":"2023-11-08T08:19:45.088734+0000","last_became_peered":"2023-11-08T08:19:45.088734+0000","last_unstale":"2023-11-08T08:19:45.090801+0000","last_undegraded":"2023-11-08T08:19:45.090801+0000","last_fullsized":"2023-11-08T08:19:45.090801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:47.846149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095266+0000","last_change":"2023-11-08T08:19:45.088048+0000","last_active":"2023-11-08T08:19:45.095266+0000","last_peered":"2023-11-08T08:19:45.095266+0000","last_clean":"2023-11-08T08:19:45.095266+0000","last_became_active":"2023-11-08T08:19:45.087764+0000","last_became_peered":"2023-11-08T08:19:45.087764+0000","last_unstale":"2023-11-08T08:19:45.095266+0000","last_undegraded":"2023-11-08T08:19:45.095266+0000","last_fullsized":"2023-11-08T08:19:45.095266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:47:30.709639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.091216+0000","last_change":"2023-11-08T08:19:45.087534+0000","last_active":"2023-11-08T08:19:45.091216+0000","last_peered":"2023-11-08T08:19:45.091216+0000","last_clean":"2023-11-08T08:19:45.091216+0000","last_became_active":"2023-11-08T08:19:45.087275+0000","last_became_peered":"2023-11-08T08:19:45.087275+0000","last_unstale":"2023-11-08T08:19:45.091216+0000","last_undegraded":"2023-11-08T08:19:45.091216+0000","last_fullsized":"2023-11-08T08:19:45.091216+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:01:12.773864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099286+0000","last_change":"2023-11-08T08:19:45.097882+0000","last_active":"2023-11-08T08:19:45.099286+0000","last_peered":"2023-11-08T08:19:45.099286+0000","last_clean":"2023-11-08T08:19:45.099286+0000","last_became_active":"2023-11-08T08:19:45.097581+0000","last_became_peered":"2023-11-08T08:19:45.097581+0000","last_unstale":"2023-11-08T08:19:45.099286+0000","last_undegraded":"2023-11-08T08:19:45.099286+0000","last_fullsized":"2023-11-08T08:19:45.099286+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:30.822774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099292+0000","last_change":"2023-11-08T08:19:45.098130+0000","last_active":"2023-11-08T08:19:45.099292+0000","last_peered":"2023-11-08T08:19:45.099292+0000","last_clean":"2023-11-08T08:19:45.099292+0000","last_became_active":"2023-11-08T08:19:45.097655+0000","last_became_peered":"2023-11-08T08:19:45.097655+0000","last_unstale":"2023-11-08T08:19:45.099292+0000","last_undegraded":"2023-11-08T08:19:45.099292+0000","last_fullsized":"2023-11-08T08:19:45.099292+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:24:01.543485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093575+0000","last_change":"2023-11-08T08:19:45.089307+0000","last_active":"2023-11-08T08:19:45.093575+0000","last_peered":"2023-11-08T08:19:45.093575+0000","last_clean":"2023-11-08T08:19:45.093575+0000","last_became_active":"2023-11-08T08:19:45.089039+0000","last_became_peered":"2023-11-08T08:19:45.089039+0000","last_unstale":"2023-11-08T08:19:45.093575+0000","last_undegraded":"2023-11-08T08:19:45.093575+0000","last_fullsized":"2023-11-08T08:19:45.093575+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:24:20.826972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.101496+0000","last_change":"2023-11-08T08:19:45.097845+0000","last_active":"2023-11-08T08:19:45.101496+0000","last_peered":"2023-11-08T08:19:45.101496+0000","last_clean":"2023-11-08T08:19:45.101496+0000","last_became_active":"2023-11-08T08:19:45.097594+0000","last_became_peered":"2023-11-08T08:19:45.097594+0000","last_unstale":"2023-11-08T08:19:45.101496+0000","last_undegraded":"2023-11-08T08:19:45.101496+0000","last_fullsized":"2023-11-08T08:19:45.101496+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:11.342423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.082880+0000","last_change":"2023-11-08T08:19:45.078263+0000","last_active":"2023-11-08T08:19:45.082880+0000","last_peered":"2023-11-08T08:19:45.082880+0000","last_clean":"2023-11-08T08:19:45.082880+0000","last_became_active":"2023-11-08T08:19:45.078025+0000","last_became_peered":"2023-11-08T08:19:45.078025+0000","last_unstale":"2023-11-08T08:19:45.082880+0000","last_undegraded":"2023-11-08T08:19:45.082880+0000","last_fullsized":"2023-11-08T08:19:45.082880+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:06:11.698122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085116+0000","last_change":"2023-11-08T08:19:45.082435+0000","last_active":"2023-11-08T08:19:45.085116+0000","last_peered":"2023-11-08T08:19:45.085116+0000","last_clean":"2023-11-08T08:19:45.085116+0000","last_became_active":"2023-11-08T08:19:45.082194+0000","last_became_peered":"2023-11-08T08:19:45.082194+0000","last_unstale":"2023-11-08T08:19:45.085116+0000","last_undegraded":"2023-11-08T08:19:45.085116+0000","last_fullsized":"2023-11-08T08:19:45.085116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:01.709942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095287+0000","last_change":"2023-11-08T08:19:45.092571+0000","last_active":"2023-11-08T08:19:45.095287+0000","last_peered":"2023-11-08T08:19:45.095287+0000","last_clean":"2023-11-08T08:19:45.095287+0000","last_became_active":"2023-11-08T08:19:45.092334+0000","last_became_peered":"2023-11-08T08:19:45.092334+0000","last_unstale":"2023-11-08T08:19:45.095287+0000","last_undegraded":"2023-11-08T08:19:45.095287+0000","last_fullsized":"2023-11-08T08:19:45.095287+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:21:15.660460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097075+0000","last_change":"2023-11-08T08:19:45.093871+0000","last_active":"2023-11-08T08:19:45.097075+0000","last_peered":"2023-11-08T08:19:45.097075+0000","last_clean":"2023-11-08T08:19:45.097075+0000","last_became_active":"2023-11-08T08:19:45.091910+0000","last_became_peered":"2023-11-08T08:19:45.091910+0000","last_unstale":"2023-11-08T08:19:45.097075+0000","last_undegraded":"2023-11-08T08:19:45.097075+0000","last_fullsized":"2023-11-08T08:19:45.097075+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:30.822774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.088838+0000","last_change":"2023-11-08T08:19:45.083839+0000","last_active":"2023-11-08T08:19:45.088838+0000","last_peered":"2023-11-08T08:19:45.088838+0000","last_clean":"2023-11-08T08:19:45.088838+0000","last_became_active":"2023-11-08T08:19:45.083587+0000","last_became_peered":"2023-11-08T08:19:45.083587+0000","last_unstale":"2023-11-08T08:19:45.088838+0000","last_undegraded":"2023-11-08T08:19:45.088838+0000","last_fullsized":"2023-11-08T08:19:45.088838+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:30:44.049974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094660+0000","last_change":"2023-11-08T08:19:45.090664+0000","last_active":"2023-11-08T08:19:45.094660+0000","last_peered":"2023-11-08T08:19:45.094660+0000","last_clean":"2023-11-08T08:19:45.094660+0000","last_became_active":"2023-11-08T08:19:45.090347+0000","last_became_peered":"2023-11-08T08:19:45.090347+0000","last_unstale":"2023-11-08T08:19:45.094660+0000","last_undegraded":"2023-11-08T08:19:45.094660+0000","last_fullsized":"2023-11-08T08:19:45.094660+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:09:49.510438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092929+0000","last_change":"2023-11-08T08:19:45.090032+0000","last_active":"2023-11-08T08:19:45.092929+0000","last_peered":"2023-11-08T08:19:45.092929+0000","last_clean":"2023-11-08T08:19:45.092929+0000","last_became_active":"2023-11-08T08:19:45.089805+0000","last_became_peered":"2023-11-08T08:19:45.089805+0000","last_unstale":"2023-11-08T08:19:45.092929+0000","last_undegraded":"2023-11-08T08:19:45.092929+0000","last_fullsized":"2023-11-08T08:19:45.092929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:13.833737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.091165+0000","last_change":"2023-11-08T08:19:45.087464+0000","last_active":"2023-11-08T08:19:45.091165+0000","last_peered":"2023-11-08T08:19:45.091165+0000","last_clean":"2023-11-08T08:19:45.091165+0000","last_became_active":"2023-11-08T08:19:45.087193+0000","last_became_peered":"2023-11-08T08:19:45.087193+0000","last_unstale":"2023-11-08T08:19:45.091165+0000","last_undegraded":"2023-11-08T08:19:45.091165+0000","last_fullsized":"2023-11-08T08:19:45.091165+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:34:20.368535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097239+0000","last_change":"2023-11-08T08:19:45.092890+0000","last_active":"2023-11-08T08:19:45.097239+0000","last_peered":"2023-11-08T08:19:45.097239+0000","last_clean":"2023-11-08T08:19:45.097239+0000","last_became_active":"2023-11-08T08:19:45.092673+0000","last_became_peered":"2023-11-08T08:19:45.092673+0000","last_unstale":"2023-11-08T08:19:45.097239+0000","last_undegraded":"2023-11-08T08:19:45.097239+0000","last_fullsized":"2023-11-08T08:19:45.097239+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:05:13.351069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097229+0000","last_change":"2023-11-08T08:19:45.093015+0000","last_active":"2023-11-08T08:19:45.097229+0000","last_peered":"2023-11-08T08:19:45.097229+0000","last_clean":"2023-11-08T08:19:45.097229+0000","last_became_active":"2023-11-08T08:19:45.092796+0000","last_became_peered":"2023-11-08T08:19:45.092796+0000","last_unstale":"2023-11-08T08:19:45.097229+0000","last_undegraded":"2023-11-08T08:19:45.097229+0000","last_fullsized":"2023-11-08T08:19:45.097229+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:31:47.050095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090219+0000","last_change":"2023-11-08T08:19:45.086914+0000","last_active":"2023-11-08T08:19:45.090219+0000","last_peered":"2023-11-08T08:19:45.090219+0000","last_clean":"2023-11-08T08:19:45.090219+0000","last_became_active":"2023-11-08T08:19:45.086672+0000","last_became_peered":"2023-11-08T08:19:45.086672+0000","last_unstale":"2023-11-08T08:19:45.090219+0000","last_undegraded":"2023-11-08T08:19:45.090219+0000","last_fullsized":"2023-11-08T08:19:45.090219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:07:47.583398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094601+0000","last_change":"2023-11-08T08:19:45.091639+0000","last_active":"2023-11-08T08:19:45.094601+0000","last_peered":"2023-11-08T08:19:45.094601+0000","last_clean":"2023-11-08T08:19:45.094601+0000","last_became_active":"2023-11-08T08:19:45.088498+0000","last_became_peered":"2023-11-08T08:19:45.088498+0000","last_unstale":"2023-11-08T08:19:45.094601+0000","last_undegraded":"2023-11-08T08:19:45.094601+0000","last_fullsized":"2023-11-08T08:19:45.094601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:48.985661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097484+0000","last_change":"2023-11-08T08:19:45.094581+0000","last_active":"2023-11-08T08:19:45.097484+0000","last_peered":"2023-11-08T08:19:45.097484+0000","last_clean":"2023-11-08T08:19:45.097484+0000","last_became_active":"2023-11-08T08:19:45.094404+0000","last_became_peered":"2023-11-08T08:19:45.094404+0000","last_unstale":"2023-11-08T08:19:45.097484+0000","last_undegraded":"2023-11-08T08:19:45.097484+0000","last_fullsized":"2023-11-08T08:19:45.097484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:20:48.892808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"31'4","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.086403+0000","last_change":"2023-11-08T08:19:44.692977+0000","last_active":"2023-11-08T08:19:45.086403+0000","last_peered":"2023-11-08T08:19:45.086403+0000","last_clean":"2023-11-08T08:19:45.086403+0000","last_became_active":"2023-11-08T08:19:44.692278+0000","last_became_peered":"2023-11-08T08:19:44.692278+0000","last_unstale":"2023-11-08T08:19:45.086403+0000","last_undegraded":"2023-11-08T08:19:45.086403+0000","last_fullsized":"2023-11-08T08:19:45.086403+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:01:35.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090647+0000","last_change":"2023-11-08T08:19:45.086526+0000","last_active":"2023-11-08T08:19:45.090647+0000","last_peered":"2023-11-08T08:19:45.090647+0000","last_clean":"2023-11-08T08:19:45.090647+0000","last_became_active":"2023-11-08T08:19:45.086071+0000","last_became_peered":"2023-11-08T08:19:45.086071+0000","last_unstale":"2023-11-08T08:19:45.090647+0000","last_undegraded":"2023-11-08T08:19:45.090647+0000","last_fullsized":"2023-11-08T08:19:45.090647+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:56:04.125796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095653+0000","last_change":"2023-11-08T08:19:44.693274+0000","last_active":"2023-11-08T08:19:45.095653+0000","last_peered":"2023-11-08T08:19:45.095653+0000","last_clean":"2023-11-08T08:19:45.095653+0000","last_became_active":"2023-11-08T08:19:44.692875+0000","last_became_peered":"2023-11-08T08:19:44.692875+0000","last_unstale":"2023-11-08T08:19:45.095653+0000","last_undegraded":"2023-11-08T08:19:45.095653+0000","last_fullsized":"2023-11-08T08:19:45.095653+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:56:37.112660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097290+0000","last_change":"2023-11-08T08:19:45.094108+0000","last_active":"2023-11-08T08:19:45.097290+0000","last_peered":"2023-11-08T08:19:45.097290+0000","last_clean":"2023-11-08T08:19:45.097290+0000","last_became_active":"2023-11-08T08:19:45.093788+0000","last_became_peered":"2023-11-08T08:19:45.093788+0000","last_unstale":"2023-11-08T08:19:45.097290+0000","last_undegraded":"2023-11-08T08:19:45.097290+0000","last_fullsized":"2023-11-08T08:19:45.097290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:32.777372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093560+0000","last_change":"2023-11-08T08:19:44.688928+0000","last_active":"2023-11-08T08:19:45.093560+0000","last_peered":"2023-11-08T08:19:45.093560+0000","last_clean":"2023-11-08T08:19:45.093560+0000","last_became_active":"2023-11-08T08:19:44.688599+0000","last_became_peered":"2023-11-08T08:19:44.688599+0000","last_unstale":"2023-11-08T08:19:45.093560+0000","last_undegraded":"2023-11-08T08:19:45.093560+0000","last_fullsized":"2023-11-08T08:19:45.093560+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:19:30.644800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089679+0000","last_change":"2023-11-08T08:19:45.085475+0000","last_active":"2023-11-08T08:19:45.089679+0000","last_peered":"2023-11-08T08:19:45.089679+0000","last_clean":"2023-11-08T08:19:45.089679+0000","last_became_active":"2023-11-08T08:19:45.085063+0000","last_became_peered":"2023-11-08T08:19:45.085063+0000","last_unstale":"2023-11-08T08:19:45.089679+0000","last_undegraded":"2023-11-08T08:19:45.089679+0000","last_fullsized":"2023-11-08T08:19:45.089679+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:30:44.049974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.086259+0000","last_change":"2023-11-08T08:19:44.694225+0000","last_active":"2023-11-08T08:19:45.086259+0000","last_peered":"2023-11-08T08:19:45.086259+0000","last_clean":"2023-11-08T08:19:45.086259+0000","last_became_active":"2023-11-08T08:19:44.693954+0000","last_became_peered":"2023-11-08T08:19:44.693954+0000","last_unstale":"2023-11-08T08:19:45.086259+0000","last_undegraded":"2023-11-08T08:19:45.086259+0000","last_fullsized":"2023-11-08T08:19:45.086259+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:53:43.166175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099615+0000","last_change":"2023-11-08T08:19:45.094474+0000","last_active":"2023-11-08T08:19:45.099615+0000","last_peered":"2023-11-08T08:19:45.099615+0000","last_clean":"2023-11-08T08:19:45.099615+0000","last_became_active":"2023-11-08T08:19:45.094144+0000","last_became_peered":"2023-11-08T08:19:45.094144+0000","last_unstale":"2023-11-08T08:19:45.099615+0000","last_undegraded":"2023-11-08T08:19:45.099615+0000","last_fullsized":"2023-11-08T08:19:45.099615+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:32.777372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087736+0000","last_change":"2023-11-08T08:19:44.692546+0000","last_active":"2023-11-08T08:19:45.087736+0000","last_peered":"2023-11-08T08:19:45.087736+0000","last_clean":"2023-11-08T08:19:45.087736+0000","last_became_active":"2023-11-08T08:19:44.692320+0000","last_became_peered":"2023-11-08T08:19:44.692320+0000","last_unstale":"2023-11-08T08:19:45.087736+0000","last_undegraded":"2023-11-08T08:19:45.087736+0000","last_fullsized":"2023-11-08T08:19:45.087736+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:38:53.026823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093066+0000","last_change":"2023-11-08T08:19:45.091345+0000","last_active":"2023-11-08T08:19:45.093066+0000","last_peered":"2023-11-08T08:19:45.093066+0000","last_clean":"2023-11-08T08:19:45.093066+0000","last_became_active":"2023-11-08T08:19:45.091173+0000","last_became_peered":"2023-11-08T08:19:45.091173+0000","last_unstale":"2023-11-08T08:19:45.093066+0000","last_undegraded":"2023-11-08T08:19:45.093066+0000","last_fullsized":"2023-11-08T08:19:45.093066+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:19:31.649628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.088305+0000","last_change":"2023-11-08T08:19:44.689843+0000","last_active":"2023-11-08T08:19:45.088305+0000","last_peered":"2023-11-08T08:19:45.088305+0000","last_clean":"2023-11-08T08:19:45.088305+0000","last_became_active":"2023-11-08T08:19:44.689246+0000","last_became_peered":"2023-11-08T08:19:44.689246+0000","last_unstale":"2023-11-08T08:19:45.088305+0000","last_undegraded":"2023-11-08T08:19:45.088305+0000","last_fullsized":"2023-11-08T08:19:45.088305+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:55.598545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094735+0000","last_change":"2023-11-08T08:19:45.085858+0000","last_active":"2023-11-08T08:19:45.094735+0000","last_peered":"2023-11-08T08:19:45.094735+0000","last_clean":"2023-11-08T08:19:45.094735+0000","last_became_active":"2023-11-08T08:19:45.085460+0000","last_became_peered":"2023-11-08T08:19:45.085460+0000","last_unstale":"2023-11-08T08:19:45.094735+0000","last_undegraded":"2023-11-08T08:19:45.094735+0000","last_fullsized":"2023-11-08T08:19:45.094735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:48.985661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.072957+0000","last_change":"2023-11-08T08:19:44.691843+0000","last_active":"2023-11-08T08:19:45.072957+0000","last_peered":"2023-11-08T08:19:45.072957+0000","last_clean":"2023-11-08T08:19:45.072957+0000","last_became_active":"2023-11-08T08:19:44.691540+0000","last_became_peered":"2023-11-08T08:19:44.691540+0000","last_unstale":"2023-11-08T08:19:45.072957+0000","last_undegraded":"2023-11-08T08:19:45.072957+0000","last_fullsized":"2023-11-08T08:19:45.072957+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:11:59.871085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092923+0000","last_change":"2023-11-08T08:19:45.091601+0000","last_active":"2023-11-08T08:19:45.092923+0000","last_peered":"2023-11-08T08:19:45.092923+0000","last_clean":"2023-11-08T08:19:45.092923+0000","last_became_active":"2023-11-08T08:19:45.091219+0000","last_became_peered":"2023-11-08T08:19:45.091219+0000","last_unstale":"2023-11-08T08:19:45.092923+0000","last_undegraded":"2023-11-08T08:19:45.092923+0000","last_fullsized":"2023-11-08T08:19:45.092923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:49:26.597569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.077241+0000","last_change":"2023-11-08T08:19:44.690129+0000","last_active":"2023-11-08T08:19:45.077241+0000","last_peered":"2023-11-08T08:19:45.077241+0000","last_clean":"2023-11-08T08:19:45.077241+0000","last_became_active":"2023-11-08T08:19:44.689829+0000","last_became_peered":"2023-11-08T08:19:44.689829+0000","last_unstale":"2023-11-08T08:19:45.077241+0000","last_undegraded":"2023-11-08T08:19:45.077241+0000","last_fullsized":"2023-11-08T08:19:45.077241+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:24:19.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094500+0000","last_change":"2023-11-08T08:19:45.089209+0000","last_active":"2023-11-08T08:19:45.094500+0000","last_peered":"2023-11-08T08:19:45.094500+0000","last_clean":"2023-11-08T08:19:45.094500+0000","last_became_active":"2023-11-08T08:19:45.088913+0000","last_became_peered":"2023-11-08T08:19:45.088913+0000","last_unstale":"2023-11-08T08:19:45.094500+0000","last_undegraded":"2023-11-08T08:19:45.094500+0000","last_fullsized":"2023-11-08T08:19:45.094500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:45:18.850871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.075123+0000","last_change":"2023-11-08T08:19:44.686234+0000","last_active":"2023-11-08T08:19:45.075123+0000","last_peered":"2023-11-08T08:19:45.075123+0000","last_clean":"2023-11-08T08:19:45.075123+0000","last_became_active":"2023-11-08T08:19:44.685928+0000","last_became_peered":"2023-11-08T08:19:44.685928+0000","last_unstale":"2023-11-08T08:19:45.075123+0000","last_undegraded":"2023-11-08T08:19:45.075123+0000","last_fullsized":"2023-11-08T08:19:45.075123+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:56:03.120968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.088736+0000","last_change":"2023-11-08T08:19:45.081453+0000","last_active":"2023-11-08T08:19:45.088736+0000","last_peered":"2023-11-08T08:19:45.088736+0000","last_clean":"2023-11-08T08:19:45.088736+0000","last_became_active":"2023-11-08T08:19:45.081186+0000","last_became_peered":"2023-11-08T08:19:45.081186+0000","last_unstale":"2023-11-08T08:19:45.088736+0000","last_undegraded":"2023-11-08T08:19:45.088736+0000","last_fullsized":"2023-11-08T08:19:45.088736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:56.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085932+0000","last_change":"2023-11-08T08:19:44.691340+0000","last_active":"2023-11-08T08:19:45.085932+0000","last_peered":"2023-11-08T08:19:45.085932+0000","last_clean":"2023-11-08T08:19:45.085932+0000","last_became_active":"2023-11-08T08:19:44.691068+0000","last_became_peered":"2023-11-08T08:19:44.691068+0000","last_unstale":"2023-11-08T08:19:45.085932+0000","last_undegraded":"2023-11-08T08:19:45.085932+0000","last_fullsized":"2023-11-08T08:19:45.085932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:47:41.135797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094589+0000","last_change":"2023-11-08T08:19:45.088539+0000","last_active":"2023-11-08T08:19:45.094589+0000","last_peered":"2023-11-08T08:19:45.094589+0000","last_clean":"2023-11-08T08:19:45.094589+0000","last_became_active":"2023-11-08T08:19:45.087962+0000","last_became_peered":"2023-11-08T08:19:45.087962+0000","last_unstale":"2023-11-08T08:19:45.094589+0000","last_undegraded":"2023-11-08T08:19:45.094589+0000","last_fullsized":"2023-11-08T08:19:45.094589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:08:06.508307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"31'3","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.098027+0000","last_change":"2023-11-08T08:19:44.689537+0000","last_active":"2023-11-08T08:19:45.098027+0000","last_peered":"2023-11-08T08:19:45.098027+0000","last_clean":"2023-11-08T08:19:45.098027+0000","last_became_active":"2023-11-08T08:19:44.689278+0000","last_became_peered":"2023-11-08T08:19:44.689278+0000","last_unstale":"2023-11-08T08:19:45.098027+0000","last_undegraded":"2023-11-08T08:19:45.098027+0000","last_fullsized":"2023-11-08T08:19:45.098027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:49:48.410399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090127+0000","last_change":"2023-11-08T08:19:45.084874+0000","last_active":"2023-11-08T08:19:45.090127+0000","last_peered":"2023-11-08T08:19:45.090127+0000","last_clean":"2023-11-08T08:19:45.090127+0000","last_became_active":"2023-11-08T08:19:45.084603+0000","last_became_peered":"2023-11-08T08:19:45.084603+0000","last_unstale":"2023-11-08T08:19:45.090127+0000","last_undegraded":"2023-11-08T08:19:45.090127+0000","last_fullsized":"2023-11-08T08:19:45.090127+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:06:11.698122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"31'6","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099967+0000","last_change":"2023-11-08T08:19:44.689549+0000","last_active":"2023-11-08T08:19:45.099967+0000","last_peered":"2023-11-08T08:19:45.099967+0000","last_clean":"2023-11-08T08:19:45.099967+0000","last_became_active":"2023-11-08T08:19:44.689293+0000","last_became_peered":"2023-11-08T08:19:44.689293+0000","last_unstale":"2023-11-08T08:19:45.099967+0000","last_undegraded":"2023-11-08T08:19:45.099967+0000","last_fullsized":"2023-11-08T08:19:45.099967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:35:38.697974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097090+0000","last_change":"2023-11-08T08:19:45.093869+0000","last_active":"2023-11-08T08:19:45.097090+0000","last_peered":"2023-11-08T08:19:45.097090+0000","last_clean":"2023-11-08T08:19:45.097090+0000","last_became_active":"2023-11-08T08:19:45.090749+0000","last_became_peered":"2023-11-08T08:19:45.090749+0000","last_unstale":"2023-11-08T08:19:45.097090+0000","last_undegraded":"2023-11-08T08:19:45.097090+0000","last_fullsized":"2023-11-08T08:19:45.097090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:20:48.892808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093888+0000","last_change":"2023-11-08T08:19:44.693148+0000","last_active":"2023-11-08T08:19:45.093888+0000","last_peered":"2023-11-08T08:19:45.093888+0000","last_clean":"2023-11-08T08:19:45.093888+0000","last_became_active":"2023-11-08T08:19:44.692887+0000","last_became_peered":"2023-11-08T08:19:44.692887+0000","last_unstale":"2023-11-08T08:19:45.093888+0000","last_undegraded":"2023-11-08T08:19:45.093888+0000","last_fullsized":"2023-11-08T08:19:45.093888+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:00.705114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089656+0000","last_change":"2023-11-08T08:19:45.084765+0000","last_active":"2023-11-08T08:19:45.089656+0000","last_peered":"2023-11-08T08:19:45.089656+0000","last_clean":"2023-11-08T08:19:45.089656+0000","last_became_active":"2023-11-08T08:19:45.084449+0000","last_became_peered":"2023-11-08T08:19:45.084449+0000","last_unstale":"2023-11-08T08:19:45.089656+0000","last_undegraded":"2023-11-08T08:19:45.089656+0000","last_fullsized":"2023-11-08T08:19:45.089656+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:12:00.875913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087428+0000","last_change":"2023-11-08T08:19:44.696841+0000","last_active":"2023-11-08T08:19:45.087428+0000","last_peered":"2023-11-08T08:19:45.087428+0000","last_clean":"2023-11-08T08:19:45.087428+0000","last_became_active":"2023-11-08T08:19:44.696606+0000","last_became_peered":"2023-11-08T08:19:44.696606+0000","last_unstale":"2023-11-08T08:19:45.087428+0000","last_undegraded":"2023-11-08T08:19:45.087428+0000","last_fullsized":"2023-11-08T08:19:45.087428+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:54:04.535006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.082650+0000","last_change":"2023-11-08T08:19:45.076678+0000","last_active":"2023-11-08T08:19:45.082650+0000","last_peered":"2023-11-08T08:19:45.082650+0000","last_clean":"2023-11-08T08:19:45.082650+0000","last_became_active":"2023-11-08T08:19:45.076478+0000","last_became_peered":"2023-11-08T08:19:45.076478+0000","last_unstale":"2023-11-08T08:19:45.082650+0000","last_undegraded":"2023-11-08T08:19:45.082650+0000","last_fullsized":"2023-11-08T08:19:45.082650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:49:26.597569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085707+0000","last_change":"2023-11-08T08:19:44.692380+0000","last_active":"2023-11-08T08:19:45.085707+0000","last_peered":"2023-11-08T08:19:45.085707+0000","last_clean":"2023-11-08T08:19:45.085707+0000","last_became_active":"2023-11-08T08:19:44.690445+0000","last_became_peered":"2023-11-08T08:19:44.690445+0000","last_unstale":"2023-11-08T08:19:45.085707+0000","last_undegraded":"2023-11-08T08:19:45.085707+0000","last_fullsized":"2023-11-08T08:19:45.085707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:12.828909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093091+0000","last_change":"2023-11-08T08:19:45.087566+0000","last_active":"2023-11-08T08:19:45.093091+0000","last_peered":"2023-11-08T08:19:45.093091+0000","last_clean":"2023-11-08T08:19:45.093091+0000","last_became_active":"2023-11-08T08:19:45.087288+0000","last_became_peered":"2023-11-08T08:19:45.087288+0000","last_unstale":"2023-11-08T08:19:45.093091+0000","last_undegraded":"2023-11-08T08:19:45.093091+0000","last_fullsized":"2023-11-08T08:19:45.093091+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:47.846149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"31'8","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090368+0000","last_change":"2023-11-08T08:19:44.690997+0000","last_active":"2023-11-08T08:19:45.090368+0000","last_peered":"2023-11-08T08:19:45.090368+0000","last_clean":"2023-11-08T08:19:45.090368+0000","last_became_active":"2023-11-08T08:19:44.690731+0000","last_became_peered":"2023-11-08T08:19:44.690731+0000","last_unstale":"2023-11-08T08:19:45.090368+0000","last_undegraded":"2023-11-08T08:19:45.090368+0000","last_fullsized":"2023-11-08T08:19:45.090368+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:04:54.456244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089074+0000","last_change":"2023-11-08T08:19:45.084220+0000","last_active":"2023-11-08T08:19:45.089074+0000","last_peered":"2023-11-08T08:19:45.089074+0000","last_clean":"2023-11-08T08:19:45.089074+0000","last_became_active":"2023-11-08T08:19:45.083895+0000","last_became_peered":"2023-11-08T08:19:45.083895+0000","last_unstale":"2023-11-08T08:19:45.089074+0000","last_undegraded":"2023-11-08T08:19:45.089074+0000","last_fullsized":"2023-11-08T08:19:45.089074+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:11.342423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.084240+0000","last_change":"2023-11-08T08:19:44.688634+0000","last_active":"2023-11-08T08:19:45.084240+0000","last_peered":"2023-11-08T08:19:45.084240+0000","last_clean":"2023-11-08T08:19:45.084240+0000","last_became_active":"2023-11-08T08:19:44.688382+0000","last_became_peered":"2023-11-08T08:19:44.688382+0000","last_unstale":"2023-11-08T08:19:45.084240+0000","last_undegraded":"2023-11-08T08:19:45.084240+0000","last_fullsized":"2023-11-08T08:19:45.084240+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:06:16.804906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092344+0000","last_change":"2023-11-08T08:19:45.088639+0000","last_active":"2023-11-08T08:19:45.092344+0000","last_peered":"2023-11-08T08:19:45.092344+0000","last_clean":"2023-11-08T08:19:45.092344+0000","last_became_active":"2023-11-08T08:19:45.088394+0000","last_became_peered":"2023-11-08T08:19:45.088394+0000","last_unstale":"2023-11-08T08:19:45.092344+0000","last_undegraded":"2023-11-08T08:19:45.092344+0000","last_fullsized":"2023-11-08T08:19:45.092344+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:01.709942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097998+0000","last_change":"2023-11-08T08:19:44.690147+0000","last_active":"2023-11-08T08:19:45.097998+0000","last_peered":"2023-11-08T08:19:45.097998+0000","last_clean":"2023-11-08T08:19:45.097998+0000","last_became_active":"2023-11-08T08:19:44.689868+0000","last_became_peered":"2023-11-08T08:19:44.689868+0000","last_unstale":"2023-11-08T08:19:45.097998+0000","last_undegraded":"2023-11-08T08:19:45.097998+0000","last_fullsized":"2023-11-08T08:19:45.097998+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:01:11.769036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090226+0000","last_change":"2023-11-08T08:19:45.086841+0000","last_active":"2023-11-08T08:19:45.090226+0000","last_peered":"2023-11-08T08:19:45.090226+0000","last_clean":"2023-11-08T08:19:45.090226+0000","last_became_active":"2023-11-08T08:19:45.086558+0000","last_became_peered":"2023-11-08T08:19:45.086558+0000","last_unstale":"2023-11-08T08:19:45.090226+0000","last_undegraded":"2023-11-08T08:19:45.090226+0000","last_fullsized":"2023-11-08T08:19:45.090226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:03:27.481450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090742+0000","last_change":"2023-11-08T08:19:44.689939+0000","last_active":"2023-11-08T08:19:45.090742+0000","last_peered":"2023-11-08T08:19:45.090742+0000","last_clean":"2023-11-08T08:19:45.090742+0000","last_became_active":"2023-11-08T08:19:44.689669+0000","last_became_peered":"2023-11-08T08:19:44.689669+0000","last_unstale":"2023-11-08T08:19:45.090742+0000","last_undegraded":"2023-11-08T08:19:45.090742+0000","last_fullsized":"2023-11-08T08:19:45.090742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:46:22.351765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094400+0000","last_change":"2023-11-08T08:19:45.090531+0000","last_active":"2023-11-08T08:19:45.094400+0000","last_peered":"2023-11-08T08:19:45.094400+0000","last_clean":"2023-11-08T08:19:45.094400+0000","last_became_active":"2023-11-08T08:19:45.090181+0000","last_became_peered":"2023-11-08T08:19:45.090181+0000","last_unstale":"2023-11-08T08:19:45.094400+0000","last_undegraded":"2023-11-08T08:19:45.094400+0000","last_fullsized":"2023-11-08T08:19:45.094400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:47.846149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.083212+0000","last_change":"2023-11-08T08:19:44.686547+0000","last_active":"2023-11-08T08:19:45.083212+0000","last_peered":"2023-11-08T08:19:45.083212+0000","last_clean":"2023-11-08T08:19:45.083212+0000","last_became_active":"2023-11-08T08:19:44.686251+0000","last_became_peered":"2023-11-08T08:19:44.686251+0000","last_unstale":"2023-11-08T08:19:45.083212+0000","last_undegraded":"2023-11-08T08:19:45.083212+0000","last_fullsized":"2023-11-08T08:19:45.083212+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:04:54.456244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095251+0000","last_change":"2023-11-08T08:19:45.088356+0000","last_active":"2023-11-08T08:19:45.095251+0000","last_peered":"2023-11-08T08:19:45.095251+0000","last_clean":"2023-11-08T08:19:45.095251+0000","last_became_active":"2023-11-08T08:19:45.087954+0000","last_became_peered":"2023-11-08T08:19:45.087954+0000","last_unstale":"2023-11-08T08:19:45.095251+0000","last_undegraded":"2023-11-08T08:19:45.095251+0000","last_fullsized":"2023-11-08T08:19:45.095251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:50:17.804798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093492+0000","last_change":"2023-11-08T08:19:44.686890+0000","last_active":"2023-11-08T08:19:45.093492+0000","last_peered":"2023-11-08T08:19:45.093492+0000","last_clean":"2023-11-08T08:19:45.093492+0000","last_became_active":"2023-11-08T08:19:44.686592+0000","last_became_peered":"2023-11-08T08:19:44.686592+0000","last_unstale":"2023-11-08T08:19:45.093492+0000","last_undegraded":"2023-11-08T08:19:45.093492+0000","last_fullsized":"2023-11-08T08:19:45.093492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:38:53.026823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093613+0000","last_change":"2023-11-08T08:19:45.087149+0000","last_active":"2023-11-08T08:19:45.093613+0000","last_peered":"2023-11-08T08:19:45.093613+0000","last_clean":"2023-11-08T08:19:45.093613+0000","last_became_active":"2023-11-08T08:19:45.086409+0000","last_became_peered":"2023-11-08T08:19:45.086409+0000","last_unstale":"2023-11-08T08:19:45.093613+0000","last_undegraded":"2023-11-08T08:19:45.093613+0000","last_fullsized":"2023-11-08T08:19:45.093613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:19:18.354270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087524+0000","last_change":"2023-11-08T08:19:44.694936+0000","last_active":"2023-11-08T08:19:45.087524+0000","last_peered":"2023-11-08T08:19:45.087524+0000","last_clean":"2023-11-08T08:19:45.087524+0000","last_became_active":"2023-11-08T08:19:44.694644+0000","last_became_peered":"2023-11-08T08:19:44.694644+0000","last_unstale":"2023-11-08T08:19:45.087524+0000","last_undegraded":"2023-11-08T08:19:45.087524+0000","last_fullsized":"2023-11-08T08:19:45.087524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:08:50.314631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.082986+0000","last_change":"2023-11-08T08:19:45.080947+0000","last_active":"2023-11-08T08:19:45.082986+0000","last_peered":"2023-11-08T08:19:45.082986+0000","last_clean":"2023-11-08T08:19:45.082986+0000","last_became_active":"2023-11-08T08:19:45.080646+0000","last_became_peered":"2023-11-08T08:19:45.080646+0000","last_unstale":"2023-11-08T08:19:45.082986+0000","last_undegraded":"2023-11-08T08:19:45.082986+0000","last_fullsized":"2023-11-08T08:19:45.082986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:12:00.875913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.076750+0000","last_change":"2023-11-08T08:19:44.691404+0000","last_active":"2023-11-08T08:19:45.076750+0000","last_peered":"2023-11-08T08:19:45.076750+0000","last_clean":"2023-11-08T08:19:45.076750+0000","last_became_active":"2023-11-08T08:19:44.691130+0000","last_became_peered":"2023-11-08T08:19:44.691130+0000","last_unstale":"2023-11-08T08:19:45.076750+0000","last_undegraded":"2023-11-08T08:19:45.076750+0000","last_fullsized":"2023-11-08T08:19:45.076750+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:46.841322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093698+0000","last_change":"2023-11-08T08:19:45.087499+0000","last_active":"2023-11-08T08:19:45.093698+0000","last_peered":"2023-11-08T08:19:45.093698+0000","last_clean":"2023-11-08T08:19:45.093698+0000","last_became_active":"2023-11-08T08:19:45.087229+0000","last_became_peered":"2023-11-08T08:19:45.087229+0000","last_unstale":"2023-11-08T08:19:45.093698+0000","last_undegraded":"2023-11-08T08:19:45.093698+0000","last_fullsized":"2023-11-08T08:19:45.093698+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:20.841622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.084758+0000","last_change":"2023-11-08T08:19:44.693768+0000","last_active":"2023-11-08T08:19:45.084758+0000","last_peered":"2023-11-08T08:19:45.084758+0000","last_clean":"2023-11-08T08:19:45.084758+0000","last_became_active":"2023-11-08T08:19:44.693483+0000","last_became_peered":"2023-11-08T08:19:44.693483+0000","last_unstale":"2023-11-08T08:19:45.084758+0000","last_undegraded":"2023-11-08T08:19:45.084758+0000","last_fullsized":"2023-11-08T08:19:45.084758+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:24:39.933425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095318+0000","last_change":"2023-11-08T08:19:45.089276+0000","last_active":"2023-11-08T08:19:45.095318+0000","last_peered":"2023-11-08T08:19:45.095318+0000","last_clean":"2023-11-08T08:19:45.095318+0000","last_became_active":"2023-11-08T08:19:45.089005+0000","last_became_peered":"2023-11-08T08:19:45.089005+0000","last_unstale":"2023-11-08T08:19:45.095318+0000","last_undegraded":"2023-11-08T08:19:45.095318+0000","last_fullsized":"2023-11-08T08:19:45.095318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:01:43.871569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085788+0000","last_change":"2023-11-08T08:19:44.690469+0000","last_active":"2023-11-08T08:19:45.085788+0000","last_peered":"2023-11-08T08:19:45.085788+0000","last_clean":"2023-11-08T08:19:45.085788+0000","last_became_active":"2023-11-08T08:19:44.690194+0000","last_became_peered":"2023-11-08T08:19:44.690194+0000","last_unstale":"2023-11-08T08:19:45.085788+0000","last_undegraded":"2023-11-08T08:19:45.085788+0000","last_fullsized":"2023-11-08T08:19:45.085788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:00.705114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.684383+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:19:44.684383+0000","last_peered":"2023-11-08T08:19:44.684383+0000","last_clean":"2023-11-08T08:19:44.684383+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:19:44.684383+0000","last_undegraded":"2023-11-08T08:19:44.684383+0000","last_fullsized":"2023-11-08T08:19:44.684383+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092851+0000","last_change":"2023-11-08T08:19:45.090352+0000","last_active":"2023-11-08T08:19:45.092851+0000","last_peered":"2023-11-08T08:19:45.092851+0000","last_clean":"2023-11-08T08:19:45.092851+0000","last_became_active":"2023-11-08T08:19:45.090100+0000","last_became_peered":"2023-11-08T08:19:45.090100+0000","last_unstale":"2023-11-08T08:19:45.092851+0000","last_undegraded":"2023-11-08T08:19:45.092851+0000","last_fullsized":"2023-11-08T08:19:45.092851+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:04:55.461072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.074440+0000","last_change":"2023-11-08T08:19:44.692860+0000","last_active":"2023-11-08T08:19:45.074440+0000","last_peered":"2023-11-08T08:19:45.074440+0000","last_clean":"2023-11-08T08:19:45.074440+0000","last_became_active":"2023-11-08T08:19:44.692585+0000","last_became_peered":"2023-11-08T08:19:44.692585+0000","last_unstale":"2023-11-08T08:19:45.074440+0000","last_undegraded":"2023-11-08T08:19:45.074440+0000","last_fullsized":"2023-11-08T08:19:45.074440+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:10.337595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.679995+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:19:44.679995+0000","last_peered":"2023-11-08T08:19:44.679995+0000","last_clean":"2023-11-08T08:19:44.679995+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:19:44.679995+0000","last_undegraded":"2023-11-08T08:19:44.679995+0000","last_fullsized":"2023-11-08T08:19:44.679995+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.083521+0000","last_change":"2023-11-08T08:19:45.081447+0000","last_active":"2023-11-08T08:19:45.083521+0000","last_peered":"2023-11-08T08:19:45.083521+0000","last_clean":"2023-11-08T08:19:45.083521+0000","last_became_active":"2023-11-08T08:19:45.081128+0000","last_became_peered":"2023-11-08T08:19:45.081128+0000","last_unstale":"2023-11-08T08:19:45.083521+0000","last_undegraded":"2023-11-08T08:19:45.083521+0000","last_fullsized":"2023-11-08T08:19:45.083521+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:06:17.809734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095573+0000","last_change":"2023-11-08T08:19:44.689594+0000","last_active":"2023-11-08T08:19:45.095573+0000","last_peered":"2023-11-08T08:19:45.095573+0000","last_clean":"2023-11-08T08:19:45.095573+0000","last_became_active":"2023-11-08T08:19:44.689351+0000","last_became_peered":"2023-11-08T08:19:44.689351+0000","last_unstale":"2023-11-08T08:19:45.095573+0000","last_undegraded":"2023-11-08T08:19:45.095573+0000","last_fullsized":"2023-11-08T08:19:45.095573+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:06:10.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.677930+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:19:44.677930+0000","last_peered":"2023-11-08T08:19:44.677930+0000","last_clean":"2023-11-08T08:19:44.677930+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:19:44.677930+0000","last_undegraded":"2023-11-08T08:19:44.677930+0000","last_fullsized":"2023-11-08T08:19:44.677930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":174,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.682134+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:19:44.682134+0000","last_peered":"2023-11-08T08:19:44.682134+0000","last_clean":"2023-11-08T08:19:44.682134+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:19:44.682134+0000","last_undegraded":"2023-11-08T08:19:44.682134+0000","last_fullsized":"2023-11-08T08:19:44.682134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090347+0000","last_change":"2023-11-08T08:19:45.083642+0000","last_active":"2023-11-08T08:19:45.090347+0000","last_peered":"2023-11-08T08:19:45.090347+0000","last_clean":"2023-11-08T08:19:45.090347+0000","last_became_active":"2023-11-08T08:19:45.083392+0000","last_became_peered":"2023-11-08T08:19:45.083392+0000","last_unstale":"2023-11-08T08:19:45.090347+0000","last_undegraded":"2023-11-08T08:19:45.090347+0000","last_fullsized":"2023-11-08T08:19:45.090347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:23:40.198997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.073647+0000","last_change":"2023-11-08T08:19:44.689997+0000","last_active":"2023-11-08T08:19:45.073647+0000","last_peered":"2023-11-08T08:19:45.073647+0000","last_clean":"2023-11-08T08:19:45.073647+0000","last_became_active":"2023-11-08T08:19:44.689657+0000","last_became_peered":"2023-11-08T08:19:44.689657+0000","last_unstale":"2023-11-08T08:19:45.073647+0000","last_undegraded":"2023-11-08T08:19:45.073647+0000","last_fullsized":"2023-11-08T08:19:45.073647+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:18:04.463027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.681188+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:19:44.681188+0000","last_peered":"2023-11-08T08:19:44.681188+0000","last_clean":"2023-11-08T08:19:44.681188+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:19:44.681188+0000","last_undegraded":"2023-11-08T08:19:44.681188+0000","last_fullsized":"2023-11-08T08:19:44.681188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095611+0000","last_change":"2023-11-08T08:19:45.092553+0000","last_active":"2023-11-08T08:19:45.095611+0000","last_peered":"2023-11-08T08:19:45.095611+0000","last_clean":"2023-11-08T08:19:45.095611+0000","last_became_active":"2023-11-08T08:19:45.092280+0000","last_became_peered":"2023-11-08T08:19:45.092280+0000","last_unstale":"2023-11-08T08:19:45.095611+0000","last_undegraded":"2023-11-08T08:19:45.095611+0000","last_fullsized":"2023-11-08T08:19:45.095611+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:08:06.508307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.069762+0000","last_change":"2023-11-08T08:19:44.688419+0000","last_active":"2023-11-08T08:19:45.069762+0000","last_peered":"2023-11-08T08:19:45.069762+0000","last_clean":"2023-11-08T08:19:45.069762+0000","last_became_active":"2023-11-08T08:19:44.688133+0000","last_became_peered":"2023-11-08T08:19:44.688133+0000","last_unstale":"2023-11-08T08:19:45.069762+0000","last_undegraded":"2023-11-08T08:19:45.069762+0000","last_fullsized":"2023-11-08T08:19:45.069762+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:54:04.535006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.684666+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:19:44.684666+0000","last_peered":"2023-11-08T08:19:44.684666+0000","last_clean":"2023-11-08T08:19:44.684666+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:19:44.684666+0000","last_undegraded":"2023-11-08T08:19:44.684666+0000","last_fullsized":"2023-11-08T08:19:44.684666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.102590+0000","last_change":"2023-11-08T08:19:45.098102+0000","last_active":"2023-11-08T08:19:45.102590+0000","last_peered":"2023-11-08T08:19:45.102590+0000","last_clean":"2023-11-08T08:19:45.102590+0000","last_became_active":"2023-11-08T08:19:45.097869+0000","last_became_peered":"2023-11-08T08:19:45.097869+0000","last_unstale":"2023-11-08T08:19:45.102590+0000","last_undegraded":"2023-11-08T08:19:45.102590+0000","last_fullsized":"2023-11-08T08:19:45.102590+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:56.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.073605+0000","last_change":"2023-11-08T08:19:44.684683+0000","last_active":"2023-11-08T08:19:45.073605+0000","last_peered":"2023-11-08T08:19:45.073605+0000","last_clean":"2023-11-08T08:19:45.073605+0000","last_became_active":"2023-11-08T08:19:44.684132+0000","last_became_peered":"2023-11-08T08:19:44.684132+0000","last_unstale":"2023-11-08T08:19:45.073605+0000","last_undegraded":"2023-11-08T08:19:45.073605+0000","last_fullsized":"2023-11-08T08:19:45.073605+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:53:10.877639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.682243+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:19:44.682243+0000","last_peered":"2023-11-08T08:19:44.682243+0000","last_clean":"2023-11-08T08:19:44.682243+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:19:44.682243+0000","last_undegraded":"2023-11-08T08:19:44.682243+0000","last_fullsized":"2023-11-08T08:19:44.682243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092944+0000","last_change":"2023-11-08T08:19:45.091802+0000","last_active":"2023-11-08T08:19:45.092944+0000","last_peered":"2023-11-08T08:19:45.092944+0000","last_clean":"2023-11-08T08:19:45.092944+0000","last_became_active":"2023-11-08T08:19:45.091517+0000","last_became_peered":"2023-11-08T08:19:45.091517+0000","last_unstale":"2023-11-08T08:19:45.092944+0000","last_undegraded":"2023-11-08T08:19:45.092944+0000","last_fullsized":"2023-11-08T08:19:45.092944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:13.833737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.069422+0000","last_change":"2023-11-08T08:19:44.692578+0000","last_active":"2023-11-08T08:19:45.069422+0000","last_peered":"2023-11-08T08:19:45.069422+0000","last_clean":"2023-11-08T08:19:45.069422+0000","last_became_active":"2023-11-08T08:19:44.692276+0000","last_became_peered":"2023-11-08T08:19:44.692276+0000","last_unstale":"2023-11-08T08:19:45.069422+0000","last_undegraded":"2023-11-08T08:19:45.069422+0000","last_fullsized":"2023-11-08T08:19:45.069422+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:08:50.314631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.687063+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:19:44.687063+0000","last_peered":"2023-11-08T08:19:44.687063+0000","last_clean":"2023-11-08T08:19:44.687063+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:19:44.687063+0000","last_undegraded":"2023-11-08T08:19:44.687063+0000","last_fullsized":"2023-11-08T08:19:44.687063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087356+0000","last_change":"2023-11-08T08:19:45.084782+0000","last_active":"2023-11-08T08:19:45.087356+0000","last_peered":"2023-11-08T08:19:45.087356+0000","last_clean":"2023-11-08T08:19:45.087356+0000","last_became_active":"2023-11-08T08:19:45.084512+0000","last_became_peered":"2023-11-08T08:19:45.084512+0000","last_unstale":"2023-11-08T08:19:45.087356+0000","last_undegraded":"2023-11-08T08:19:45.087356+0000","last_fullsized":"2023-11-08T08:19:45.087356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:38:54.031651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.074606+0000","last_change":"2023-11-08T08:19:44.693135+0000","last_active":"2023-11-08T08:19:45.074606+0000","last_peered":"2023-11-08T08:19:45.074606+0000","last_clean":"2023-11-08T08:19:45.074606+0000","last_became_active":"2023-11-08T08:19:44.692681+0000","last_became_peered":"2023-11-08T08:19:44.692681+0000","last_unstale":"2023-11-08T08:19:45.074606+0000","last_undegraded":"2023-11-08T08:19:45.074606+0000","last_fullsized":"2023-11-08T08:19:45.074606+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:12.828909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.686519+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:19:44.686519+0000","last_peered":"2023-11-08T08:19:44.686519+0000","last_clean":"2023-11-08T08:19:44.686519+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:19:44.686519+0000","last_undegraded":"2023-11-08T08:19:44.686519+0000","last_fullsized":"2023-11-08T08:19:44.686519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8224,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363616,"statfs":{"total":96636764160,"available":96628342784,"internally_reserved":0,"allocated":352256,"data_stored":140604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.59699999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":5.8410000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.623}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.433}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51800000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8268,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363572,"statfs":{"total":96636764160,"available":96628297728,"internally_reserved":0,"allocated":331776,"data_stored":132145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45500000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.58499999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.65900000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8196,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363644,"statfs":{"total":96636764160,"available":96628371456,"internally_reserved":0,"allocated":323584,"data_stored":126778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.246},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.32900000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55900000000000005}]},{"osd":6,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55100000000000005}]},{"osd":7,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.48799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8356,"kb_used_data":348,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363484,"statfs":{"total":96636764160,"available":96628207616,"internally_reserved":0,"allocated":356352,"data_stored":140626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.34399999999999997}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.49399999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.46100000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8364,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363476,"statfs":{"total":96636764160,"available":96628199424,"internally_reserved":0,"allocated":364544,"data_stored":142648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.19},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.31900000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.307},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"last":0.245}]},{"osd":4,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.495}]},{"osd":5,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51000000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.58099999999999996}]},{"osd":7,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9508,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362332,"statfs":{"total":96636764160,"available":96627027968,"internally_reserved":0,"allocated":946176,"data_stored":730992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.247},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.372}]},{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.5070000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.313}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.28699999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.69999999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.68799999999999994}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.80100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8432,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363408,"statfs":{"total":96636764160,"available":96628129792,"internally_reserved":0,"allocated":368640,"data_stored":142206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.5}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.58399999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40500000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.38200000000000001}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9444,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362396,"statfs":{"total":96636764160,"available":96627093504,"internally_reserved":0,"allocated":946176,"data_stored":731468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.6790000000000003,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.7270000000000003,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.45100000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.501},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":7.6079999999999997,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.0209999999999999,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.373}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":9.7550000000000008,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.5489999999999999,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.52400000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.35999999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:19:49.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:19:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.289+0000 7f7fa3581700 1 Processor -- start 2023-11-08T08:19:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa3581700 1 -- start start 2023-11-08T08:19:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa3581700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4660 0x7f7f9c071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa3581700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f9c0cff90 0x7f7f9c06b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa3581700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c0cfb00 con 0x7f7f9c0b4a30 2023-11-08T08:19:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa3581700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9c15aa20 con 0x7f7f9c0cff90 2023-11-08T08:19:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa3581700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c0cf4d0 con 0x7f7f9c0b4660 2023-11-08T08:19:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa0b1c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f9c0cff90 0x7f7f9c06b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa1b1e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7f9c0b4a30 0x7f7f9c0ac1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36558/0 (socket says 172.21.15.116:36558) 2023-11-08T08:19:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.290+0000 7f7fa1b1e700 1 -- 172.21.15.116:0/3257966499 learned_addr learned my addr 172.21.15.116:0/3257966499 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7fa131d700 1 --2- 172.21.15.116:0/3257966499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4660 0x7f7f9c071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7fa0b1c700 1 -- 172.21.15.116:0/3257966499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4660 msgr2=0x7f7f9c071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7fa0b1c700 1 --2- 172.21.15.116:0/3257966499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4660 0x7f7f9c071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7fa0b1c700 1 -- 172.21.15.116:0/3257966499 >> v1:172.21.15.116:6789/0 conn(0x7f7f9c0b4a30 legacy=0x7f7f9c0ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7f93fff700 1 -- 172.21.15.116:0/3257966499 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170236611 0 0) 0x7f7f9c0cfb00 con 0x7f7f9c0b4a30 2023-11-08T08:19:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7fa0b1c700 1 -- 172.21.15.116:0/3257966499 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c072550 con 0x7f7f9c0cff90 2023-11-08T08:19:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7fa0b1c700 1 --2- 172.21.15.116:0/3257966499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c06b590 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7f98009840 tx=0x7f7f98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6421bb6cb4aa61d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7f93fff700 1 -- 172.21.15.116:0/3257966499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98011830 con 0x7f7f9c0cff90 2023-11-08T08:19:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7f93fff700 1 -- 172.21.15.116:0/3257966499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f98011990 con 0x7f7f9c0cff90 2023-11-08T08:19:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.291+0000 7f7f93fff700 1 -- 172.21.15.116:0/3257966499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98011d20 con 0x7f7f9c0cff90 2023-11-08T08:19:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 -- 172.21.15.116:0/3257966499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0cff90 msgr2=0x7f7f9c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 --2- 172.21.15.116:0/3257966499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c06b590 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7f98009840 tx=0x7f7f98009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 -- 172.21.15.116:0/3257966499 shutdown_connections 2023-11-08T08:19:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 --2- 172.21.15.116:0/3257966499 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4660 0x7f7f9c071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 --2- 172.21.15.116:0/3257966499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c06b590 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 -- 172.21.15.116:0/3257966499 >> 172.21.15.116:0/3257966499 conn(0x7f7f9c0b2fc0 msgr2=0x7f7f9c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 -- 172.21.15.116:0/3257966499 shutdown_connections 2023-11-08T08:19:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.292+0000 7f7fa3581700 1 -- 172.21.15.116:0/3257966499 wait complete. 2023-11-08T08:19:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.293+0000 7f7fa3581700 1 Processor -- start 2023-11-08T08:19:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.293+0000 7f7fa3581700 1 -- start start 2023-11-08T08:19:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 0x7f7f9c0b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4a30 0x7f7f9c152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c0ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c0caeb0 con 0x7f7f9c0b4660 2023-11-08T08:19:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c0c41a0 con 0x7f7f9c0cff90 2023-11-08T08:19:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c0c4480 con 0x7f7f9c0b4a30 2023-11-08T08:19:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 0x7f7f9c0b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 0x7f7f9c0b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60246/0 (socket says 172.21.15.116:60246) 2023-11-08T08:19:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 -- 172.21.15.116:0/4055055034 learned_addr learned my addr 172.21.15.116:0/4055055034 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa1b1e700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c0ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa0b1c700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4a30 0x7f7f9c152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 -- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4a30 msgr2=0x7f7f9c152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4a30 0x7f7f9c152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 -- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f9c0cff90 msgr2=0x7f7f9c0ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c0ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 -- 172.21.15.116:0/4055055034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980094a0 con 0x7f7f9c0b4660 2023-11-08T08:19:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa1b1e700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c0ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa0b1c700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4a30 0x7f7f9c152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa131d700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 0x7f7f9c0b00a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e810 tx=0x7f7f8c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c013610 con 0x7f7f9c0b4660 2023-11-08T08:19:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c013770 con 0x7f7f9c0b4660 2023-11-08T08:19:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9c06bad0 con 0x7f7f9c0b4660 2023-11-08T08:19:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.294+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c013b20 con 0x7f7f9c0b4660 2023-11-08T08:19:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.295+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9c0cf790 con 0x7f7f9c0b4660 2023-11-08T08:19:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.296+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9c0b4eb0 con 0x7f7f9c0b4660 2023-11-08T08:19:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.296+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f8c023070 con 0x7f7f9c0b4660 2023-11-08T08:19:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.296+0000 7f7f91ffb700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7f800682f0 0x7f7f8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.296+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7f8c085ab0 con 0x7f7f9c0b4660 2023-11-08T08:19:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.296+0000 7f7fa0b1c700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7f800682f0 0x7f7f8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.297+0000 7f7fa0b1c700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7f800682f0 0x7f7f8006a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f980090b0 tx=0x7f7f9801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.301+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f8c052220 con 0x7f7f9c0b4660 2023-11-08T08:19:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.478+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f9c0743f0 con 0x7f7f800682f0 2023-11-08T08:19:49.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.487+0000 7f7f91ffb700 1 -- 172.21.15.116:0/4055055034 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311434 (secure 0 0 0) 0x7f7f9c0743f0 con 0x7f7f800682f0 2023-11-08T08:19:49.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7f800682f0 msgr2=0x7f7f8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7f800682f0 0x7f7f8006a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f980090b0 tx=0x7f7f9801c040 comp rx=0 tx=0).stop 2023-11-08T08:19:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 msgr2=0x7f7f9c0b00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 0x7f7f9c0b00a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e810 tx=0x7f7f8c00ebd0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 shutdown_connections 2023-11-08T08:19:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7f800682f0 0x7f7f8006a7a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f9c0cff90 0x7f7f9c0ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9c0b4660 0x7f7f9c0b00a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 --2- 172.21.15.116:0/4055055034 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7f9c0b4a30 0x7f7f9c152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 >> 172.21.15.116:0/4055055034 conn(0x7f7f9c0b2fc0 msgr2=0x7f7f9c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 shutdown_connections 2023-11-08T08:19:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.490+0000 7f7fa3581700 1 -- 172.21.15.116:0/4055055034 wait complete. 2023-11-08T08:19:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:19:49.502 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2023-11-08T08:19:48.365903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68792,"kb_used_data":3896,"kb_used_omap":0,"kb_used_meta":64832,"kb_avail":754905928,"statfs":{"total":773094113280,"available":773023670272,"internally_reserved":0,"allocated":3989504,"data_stored":2287467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66387968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.698305"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089268+0000","last_change":"2023-11-08T08:19:45.084959+0000","last_active":"2023-11-08T08:19:45.089268+0000","last_peered":"2023-11-08T08:19:45.089268+0000","last_clean":"2023-11-08T08:19:45.089268+0000","last_became_active":"2023-11-08T08:19:45.084174+0000","last_became_peered":"2023-11-08T08:19:45.084174+0000","last_unstale":"2023-11-08T08:19:45.089268+0000","last_undegraded":"2023-11-08T08:19:45.089268+0000","last_fullsized":"2023-11-08T08:19:45.089268+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:05:13.351069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089766+0000","last_change":"2023-11-08T08:19:45.084411+0000","last_active":"2023-11-08T08:19:45.089766+0000","last_peered":"2023-11-08T08:19:45.089766+0000","last_clean":"2023-11-08T08:19:45.089766+0000","last_became_active":"2023-11-08T08:19:45.084069+0000","last_became_peered":"2023-11-08T08:19:45.084069+0000","last_unstale":"2023-11-08T08:19:45.089766+0000","last_undegraded":"2023-11-08T08:19:45.089766+0000","last_fullsized":"2023-11-08T08:19:45.089766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:19:31.649628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085060+0000","last_change":"2023-11-08T08:19:45.080611+0000","last_active":"2023-11-08T08:19:45.085060+0000","last_peered":"2023-11-08T08:19:45.085060+0000","last_clean":"2023-11-08T08:19:45.085060+0000","last_became_active":"2023-11-08T08:19:45.080146+0000","last_became_peered":"2023-11-08T08:19:45.080146+0000","last_unstale":"2023-11-08T08:19:45.085060+0000","last_undegraded":"2023-11-08T08:19:45.085060+0000","last_fullsized":"2023-11-08T08:19:45.085060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:01:43.871569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.100002+0000","last_change":"2023-11-08T08:19:45.097923+0000","last_active":"2023-11-08T08:19:45.100002+0000","last_peered":"2023-11-08T08:19:45.100002+0000","last_clean":"2023-11-08T08:19:45.100002+0000","last_became_active":"2023-11-08T08:19:45.097685+0000","last_became_peered":"2023-11-08T08:19:45.097685+0000","last_unstale":"2023-11-08T08:19:45.100002+0000","last_undegraded":"2023-11-08T08:19:45.100002+0000","last_fullsized":"2023-11-08T08:19:45.100002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:01:17.819675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.083633+0000","last_change":"2023-11-08T08:19:45.081164+0000","last_active":"2023-11-08T08:19:45.083633+0000","last_peered":"2023-11-08T08:19:45.083633+0000","last_clean":"2023-11-08T08:19:45.083633+0000","last_became_active":"2023-11-08T08:19:45.080885+0000","last_became_peered":"2023-11-08T08:19:45.080885+0000","last_unstale":"2023-11-08T08:19:45.083633+0000","last_undegraded":"2023-11-08T08:19:45.083633+0000","last_fullsized":"2023-11-08T08:19:45.083633+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:48:47.154956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093695+0000","last_change":"2023-11-08T08:19:45.092153+0000","last_active":"2023-11-08T08:19:45.093695+0000","last_peered":"2023-11-08T08:19:45.093695+0000","last_clean":"2023-11-08T08:19:45.093695+0000","last_became_active":"2023-11-08T08:19:45.091883+0000","last_became_peered":"2023-11-08T08:19:45.091883+0000","last_unstale":"2023-11-08T08:19:45.093695+0000","last_undegraded":"2023-11-08T08:19:45.093695+0000","last_fullsized":"2023-11-08T08:19:45.093695+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:35:39.702801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089888+0000","last_change":"2023-11-08T08:19:45.084859+0000","last_active":"2023-11-08T08:19:45.089888+0000","last_peered":"2023-11-08T08:19:45.089888+0000","last_clean":"2023-11-08T08:19:45.089888+0000","last_became_active":"2023-11-08T08:19:45.084551+0000","last_became_peered":"2023-11-08T08:19:45.084551+0000","last_unstale":"2023-11-08T08:19:45.089888+0000","last_undegraded":"2023-11-08T08:19:45.089888+0000","last_fullsized":"2023-11-08T08:19:45.089888+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:48:10.215458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089547+0000","last_change":"2023-11-08T08:19:45.086738+0000","last_active":"2023-11-08T08:19:45.089547+0000","last_peered":"2023-11-08T08:19:45.089547+0000","last_clean":"2023-11-08T08:19:45.089547+0000","last_became_active":"2023-11-08T08:19:45.086426+0000","last_became_peered":"2023-11-08T08:19:45.086426+0000","last_unstale":"2023-11-08T08:19:45.089547+0000","last_undegraded":"2023-11-08T08:19:45.089547+0000","last_fullsized":"2023-11-08T08:19:45.089547+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:24:01.543485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093211+0000","last_change":"2023-11-08T08:19:45.089401+0000","last_active":"2023-11-08T08:19:45.093211+0000","last_peered":"2023-11-08T08:19:45.093211+0000","last_clean":"2023-11-08T08:19:45.093211+0000","last_became_active":"2023-11-08T08:19:45.089148+0000","last_became_peered":"2023-11-08T08:19:45.089148+0000","last_unstale":"2023-11-08T08:19:45.093211+0000","last_undegraded":"2023-11-08T08:19:45.093211+0000","last_fullsized":"2023-11-08T08:19:45.093211+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:24:20.826972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097370+0000","last_change":"2023-11-08T08:19:45.095504+0000","last_active":"2023-11-08T08:19:45.097370+0000","last_peered":"2023-11-08T08:19:45.097370+0000","last_clean":"2023-11-08T08:19:45.097370+0000","last_became_active":"2023-11-08T08:19:45.095233+0000","last_became_peered":"2023-11-08T08:19:45.095233+0000","last_unstale":"2023-11-08T08:19:45.097370+0000","last_undegraded":"2023-11-08T08:19:45.097370+0000","last_fullsized":"2023-11-08T08:19:45.097370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:14:20.829614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092909+0000","last_change":"2023-11-08T08:19:45.086885+0000","last_active":"2023-11-08T08:19:45.092909+0000","last_peered":"2023-11-08T08:19:45.092909+0000","last_clean":"2023-11-08T08:19:45.092909+0000","last_became_active":"2023-11-08T08:19:45.086645+0000","last_became_peered":"2023-11-08T08:19:45.086645+0000","last_unstale":"2023-11-08T08:19:45.092909+0000","last_undegraded":"2023-11-08T08:19:45.092909+0000","last_fullsized":"2023-11-08T08:19:45.092909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:35:39.702801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087482+0000","last_change":"2023-11-08T08:19:45.085758+0000","last_active":"2023-11-08T08:19:45.087482+0000","last_peered":"2023-11-08T08:19:45.087482+0000","last_clean":"2023-11-08T08:19:45.087482+0000","last_became_active":"2023-11-08T08:19:45.085129+0000","last_became_peered":"2023-11-08T08:19:45.085129+0000","last_unstale":"2023-11-08T08:19:45.087482+0000","last_undegraded":"2023-11-08T08:19:45.087482+0000","last_fullsized":"2023-11-08T08:19:45.087482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:56.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097382+0000","last_change":"2023-11-08T08:19:45.089179+0000","last_active":"2023-11-08T08:19:45.097382+0000","last_peered":"2023-11-08T08:19:45.097382+0000","last_clean":"2023-11-08T08:19:45.097382+0000","last_became_active":"2023-11-08T08:19:45.088938+0000","last_became_peered":"2023-11-08T08:19:45.088938+0000","last_unstale":"2023-11-08T08:19:45.097382+0000","last_undegraded":"2023-11-08T08:19:45.097382+0000","last_fullsized":"2023-11-08T08:19:45.097382+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:25:03.665342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090801+0000","last_change":"2023-11-08T08:19:45.089016+0000","last_active":"2023-11-08T08:19:45.090801+0000","last_peered":"2023-11-08T08:19:45.090801+0000","last_clean":"2023-11-08T08:19:45.090801+0000","last_became_active":"2023-11-08T08:19:45.088734+0000","last_became_peered":"2023-11-08T08:19:45.088734+0000","last_unstale":"2023-11-08T08:19:45.090801+0000","last_undegraded":"2023-11-08T08:19:45.090801+0000","last_fullsized":"2023-11-08T08:19:45.090801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:47.846149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095266+0000","last_change":"2023-11-08T08:19:45.088048+0000","last_active":"2023-11-08T08:19:45.095266+0000","last_peered":"2023-11-08T08:19:45.095266+0000","last_clean":"2023-11-08T08:19:45.095266+0000","last_became_active":"2023-11-08T08:19:45.087764+0000","last_became_peered":"2023-11-08T08:19:45.087764+0000","last_unstale":"2023-11-08T08:19:45.095266+0000","last_undegraded":"2023-11-08T08:19:45.095266+0000","last_fullsized":"2023-11-08T08:19:45.095266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:47:30.709639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.091216+0000","last_change":"2023-11-08T08:19:45.087534+0000","last_active":"2023-11-08T08:19:45.091216+0000","last_peered":"2023-11-08T08:19:45.091216+0000","last_clean":"2023-11-08T08:19:45.091216+0000","last_became_active":"2023-11-08T08:19:45.087275+0000","last_became_peered":"2023-11-08T08:19:45.087275+0000","last_unstale":"2023-11-08T08:19:45.091216+0000","last_undegraded":"2023-11-08T08:19:45.091216+0000","last_fullsized":"2023-11-08T08:19:45.091216+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:01:12.773864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099286+0000","last_change":"2023-11-08T08:19:45.097882+0000","last_active":"2023-11-08T08:19:45.099286+0000","last_peered":"2023-11-08T08:19:45.099286+0000","last_clean":"2023-11-08T08:19:45.099286+0000","last_became_active":"2023-11-08T08:19:45.097581+0000","last_became_peered":"2023-11-08T08:19:45.097581+0000","last_unstale":"2023-11-08T08:19:45.099286+0000","last_undegraded":"2023-11-08T08:19:45.099286+0000","last_fullsized":"2023-11-08T08:19:45.099286+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:30.822774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099292+0000","last_change":"2023-11-08T08:19:45.098130+0000","last_active":"2023-11-08T08:19:45.099292+0000","last_peered":"2023-11-08T08:19:45.099292+0000","last_clean":"2023-11-08T08:19:45.099292+0000","last_became_active":"2023-11-08T08:19:45.097655+0000","last_became_peered":"2023-11-08T08:19:45.097655+0000","last_unstale":"2023-11-08T08:19:45.099292+0000","last_undegraded":"2023-11-08T08:19:45.099292+0000","last_fullsized":"2023-11-08T08:19:45.099292+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:24:01.543485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093575+0000","last_change":"2023-11-08T08:19:45.089307+0000","last_active":"2023-11-08T08:19:45.093575+0000","last_peered":"2023-11-08T08:19:45.093575+0000","last_clean":"2023-11-08T08:19:45.093575+0000","last_became_active":"2023-11-08T08:19:45.089039+0000","last_became_peered":"2023-11-08T08:19:45.089039+0000","last_unstale":"2023-11-08T08:19:45.093575+0000","last_undegraded":"2023-11-08T08:19:45.093575+0000","last_fullsized":"2023-11-08T08:19:45.093575+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:24:20.826972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.101496+0000","last_change":"2023-11-08T08:19:45.097845+0000","last_active":"2023-11-08T08:19:45.101496+0000","last_peered":"2023-11-08T08:19:45.101496+0000","last_clean":"2023-11-08T08:19:45.101496+0000","last_became_active":"2023-11-08T08:19:45.097594+0000","last_became_peered":"2023-11-08T08:19:45.097594+0000","last_unstale":"2023-11-08T08:19:45.101496+0000","last_undegraded":"2023-11-08T08:19:45.101496+0000","last_fullsized":"2023-11-08T08:19:45.101496+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:11.342423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.082880+0000","last_change":"2023-11-08T08:19:45.078263+0000","last_active":"2023-11-08T08:19:45.082880+0000","last_peered":"2023-11-08T08:19:45.082880+0000","last_clean":"2023-11-08T08:19:45.082880+0000","last_became_active":"2023-11-08T08:19:45.078025+0000","last_became_peered":"2023-11-08T08:19:45.078025+0000","last_unstale":"2023-11-08T08:19:45.082880+0000","last_undegraded":"2023-11-08T08:19:45.082880+0000","last_fullsized":"2023-11-08T08:19:45.082880+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:06:11.698122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085116+0000","last_change":"2023-11-08T08:19:45.082435+0000","last_active":"2023-11-08T08:19:45.085116+0000","last_peered":"2023-11-08T08:19:45.085116+0000","last_clean":"2023-11-08T08:19:45.085116+0000","last_became_active":"2023-11-08T08:19:45.082194+0000","last_became_peered":"2023-11-08T08:19:45.082194+0000","last_unstale":"2023-11-08T08:19:45.085116+0000","last_undegraded":"2023-11-08T08:19:45.085116+0000","last_fullsized":"2023-11-08T08:19:45.085116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:01.709942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095287+0000","last_change":"2023-11-08T08:19:45.092571+0000","last_active":"2023-11-08T08:19:45.095287+0000","last_peered":"2023-11-08T08:19:45.095287+0000","last_clean":"2023-11-08T08:19:45.095287+0000","last_became_active":"2023-11-08T08:19:45.092334+0000","last_became_peered":"2023-11-08T08:19:45.092334+0000","last_unstale":"2023-11-08T08:19:45.095287+0000","last_undegraded":"2023-11-08T08:19:45.095287+0000","last_fullsized":"2023-11-08T08:19:45.095287+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:21:15.660460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097075+0000","last_change":"2023-11-08T08:19:45.093871+0000","last_active":"2023-11-08T08:19:45.097075+0000","last_peered":"2023-11-08T08:19:45.097075+0000","last_clean":"2023-11-08T08:19:45.097075+0000","last_became_active":"2023-11-08T08:19:45.091910+0000","last_became_peered":"2023-11-08T08:19:45.091910+0000","last_unstale":"2023-11-08T08:19:45.097075+0000","last_undegraded":"2023-11-08T08:19:45.097075+0000","last_fullsized":"2023-11-08T08:19:45.097075+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:30.822774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.088838+0000","last_change":"2023-11-08T08:19:45.083839+0000","last_active":"2023-11-08T08:19:45.088838+0000","last_peered":"2023-11-08T08:19:45.088838+0000","last_clean":"2023-11-08T08:19:45.088838+0000","last_became_active":"2023-11-08T08:19:45.083587+0000","last_became_peered":"2023-11-08T08:19:45.083587+0000","last_unstale":"2023-11-08T08:19:45.088838+0000","last_undegraded":"2023-11-08T08:19:45.088838+0000","last_fullsized":"2023-11-08T08:19:45.088838+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:30:44.049974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094660+0000","last_change":"2023-11-08T08:19:45.090664+0000","last_active":"2023-11-08T08:19:45.094660+0000","last_peered":"2023-11-08T08:19:45.094660+0000","last_clean":"2023-11-08T08:19:45.094660+0000","last_became_active":"2023-11-08T08:19:45.090347+0000","last_became_peered":"2023-11-08T08:19:45.090347+0000","last_unstale":"2023-11-08T08:19:45.094660+0000","last_undegraded":"2023-11-08T08:19:45.094660+0000","last_fullsized":"2023-11-08T08:19:45.094660+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:09:49.510438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092929+0000","last_change":"2023-11-08T08:19:45.090032+0000","last_active":"2023-11-08T08:19:45.092929+0000","last_peered":"2023-11-08T08:19:45.092929+0000","last_clean":"2023-11-08T08:19:45.092929+0000","last_became_active":"2023-11-08T08:19:45.089805+0000","last_became_peered":"2023-11-08T08:19:45.089805+0000","last_unstale":"2023-11-08T08:19:45.092929+0000","last_undegraded":"2023-11-08T08:19:45.092929+0000","last_fullsized":"2023-11-08T08:19:45.092929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:13.833737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.091165+0000","last_change":"2023-11-08T08:19:45.087464+0000","last_active":"2023-11-08T08:19:45.091165+0000","last_peered":"2023-11-08T08:19:45.091165+0000","last_clean":"2023-11-08T08:19:45.091165+0000","last_became_active":"2023-11-08T08:19:45.087193+0000","last_became_peered":"2023-11-08T08:19:45.087193+0000","last_unstale":"2023-11-08T08:19:45.091165+0000","last_undegraded":"2023-11-08T08:19:45.091165+0000","last_fullsized":"2023-11-08T08:19:45.091165+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:34:20.368535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097239+0000","last_change":"2023-11-08T08:19:45.092890+0000","last_active":"2023-11-08T08:19:45.097239+0000","last_peered":"2023-11-08T08:19:45.097239+0000","last_clean":"2023-11-08T08:19:45.097239+0000","last_became_active":"2023-11-08T08:19:45.092673+0000","last_became_peered":"2023-11-08T08:19:45.092673+0000","last_unstale":"2023-11-08T08:19:45.097239+0000","last_undegraded":"2023-11-08T08:19:45.097239+0000","last_fullsized":"2023-11-08T08:19:45.097239+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:05:13.351069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097229+0000","last_change":"2023-11-08T08:19:45.093015+0000","last_active":"2023-11-08T08:19:45.097229+0000","last_peered":"2023-11-08T08:19:45.097229+0000","last_clean":"2023-11-08T08:19:45.097229+0000","last_became_active":"2023-11-08T08:19:45.092796+0000","last_became_peered":"2023-11-08T08:19:45.092796+0000","last_unstale":"2023-11-08T08:19:45.097229+0000","last_undegraded":"2023-11-08T08:19:45.097229+0000","last_fullsized":"2023-11-08T08:19:45.097229+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:31:47.050095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090219+0000","last_change":"2023-11-08T08:19:45.086914+0000","last_active":"2023-11-08T08:19:45.090219+0000","last_peered":"2023-11-08T08:19:45.090219+0000","last_clean":"2023-11-08T08:19:45.090219+0000","last_became_active":"2023-11-08T08:19:45.086672+0000","last_became_peered":"2023-11-08T08:19:45.086672+0000","last_unstale":"2023-11-08T08:19:45.090219+0000","last_undegraded":"2023-11-08T08:19:45.090219+0000","last_fullsized":"2023-11-08T08:19:45.090219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:07:47.583398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094601+0000","last_change":"2023-11-08T08:19:45.091639+0000","last_active":"2023-11-08T08:19:45.094601+0000","last_peered":"2023-11-08T08:19:45.094601+0000","last_clean":"2023-11-08T08:19:45.094601+0000","last_became_active":"2023-11-08T08:19:45.088498+0000","last_became_peered":"2023-11-08T08:19:45.088498+0000","last_unstale":"2023-11-08T08:19:45.094601+0000","last_undegraded":"2023-11-08T08:19:45.094601+0000","last_fullsized":"2023-11-08T08:19:45.094601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:48.985661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097484+0000","last_change":"2023-11-08T08:19:45.094581+0000","last_active":"2023-11-08T08:19:45.097484+0000","last_peered":"2023-11-08T08:19:45.097484+0000","last_clean":"2023-11-08T08:19:45.097484+0000","last_became_active":"2023-11-08T08:19:45.094404+0000","last_became_peered":"2023-11-08T08:19:45.094404+0000","last_unstale":"2023-11-08T08:19:45.097484+0000","last_undegraded":"2023-11-08T08:19:45.097484+0000","last_fullsized":"2023-11-08T08:19:45.097484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:20:48.892808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"31'4","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.086403+0000","last_change":"2023-11-08T08:19:44.692977+0000","last_active":"2023-11-08T08:19:45.086403+0000","last_peered":"2023-11-08T08:19:45.086403+0000","last_clean":"2023-11-08T08:19:45.086403+0000","last_became_active":"2023-11-08T08:19:44.692278+0000","last_became_peered":"2023-11-08T08:19:44.692278+0000","last_unstale":"2023-11-08T08:19:45.086403+0000","last_undegraded":"2023-11-08T08:19:45.086403+0000","last_fullsized":"2023-11-08T08:19:45.086403+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:01:35.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090647+0000","last_change":"2023-11-08T08:19:45.086526+0000","last_active":"2023-11-08T08:19:45.090647+0000","last_peered":"2023-11-08T08:19:45.090647+0000","last_clean":"2023-11-08T08:19:45.090647+0000","last_became_active":"2023-11-08T08:19:45.086071+0000","last_became_peered":"2023-11-08T08:19:45.086071+0000","last_unstale":"2023-11-08T08:19:45.090647+0000","last_undegraded":"2023-11-08T08:19:45.090647+0000","last_fullsized":"2023-11-08T08:19:45.090647+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:56:04.125796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095653+0000","last_change":"2023-11-08T08:19:44.693274+0000","last_active":"2023-11-08T08:19:45.095653+0000","last_peered":"2023-11-08T08:19:45.095653+0000","last_clean":"2023-11-08T08:19:45.095653+0000","last_became_active":"2023-11-08T08:19:44.692875+0000","last_became_peered":"2023-11-08T08:19:44.692875+0000","last_unstale":"2023-11-08T08:19:45.095653+0000","last_undegraded":"2023-11-08T08:19:45.095653+0000","last_fullsized":"2023-11-08T08:19:45.095653+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:56:37.112660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097290+0000","last_change":"2023-11-08T08:19:45.094108+0000","last_active":"2023-11-08T08:19:45.097290+0000","last_peered":"2023-11-08T08:19:45.097290+0000","last_clean":"2023-11-08T08:19:45.097290+0000","last_became_active":"2023-11-08T08:19:45.093788+0000","last_became_peered":"2023-11-08T08:19:45.093788+0000","last_unstale":"2023-11-08T08:19:45.097290+0000","last_undegraded":"2023-11-08T08:19:45.097290+0000","last_fullsized":"2023-11-08T08:19:45.097290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:32.777372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093560+0000","last_change":"2023-11-08T08:19:44.688928+0000","last_active":"2023-11-08T08:19:45.093560+0000","last_peered":"2023-11-08T08:19:45.093560+0000","last_clean":"2023-11-08T08:19:45.093560+0000","last_became_active":"2023-11-08T08:19:44.688599+0000","last_became_peered":"2023-11-08T08:19:44.688599+0000","last_unstale":"2023-11-08T08:19:45.093560+0000","last_undegraded":"2023-11-08T08:19:45.093560+0000","last_fullsized":"2023-11-08T08:19:45.093560+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:19:30.644800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089679+0000","last_change":"2023-11-08T08:19:45.085475+0000","last_active":"2023-11-08T08:19:45.089679+0000","last_peered":"2023-11-08T08:19:45.089679+0000","last_clean":"2023-11-08T08:19:45.089679+0000","last_became_active":"2023-11-08T08:19:45.085063+0000","last_became_peered":"2023-11-08T08:19:45.085063+0000","last_unstale":"2023-11-08T08:19:45.089679+0000","last_undegraded":"2023-11-08T08:19:45.089679+0000","last_fullsized":"2023-11-08T08:19:45.089679+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:30:44.049974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.086259+0000","last_change":"2023-11-08T08:19:44.694225+0000","last_active":"2023-11-08T08:19:45.086259+0000","last_peered":"2023-11-08T08:19:45.086259+0000","last_clean":"2023-11-08T08:19:45.086259+0000","last_became_active":"2023-11-08T08:19:44.693954+0000","last_became_peered":"2023-11-08T08:19:44.693954+0000","last_unstale":"2023-11-08T08:19:45.086259+0000","last_undegraded":"2023-11-08T08:19:45.086259+0000","last_fullsized":"2023-11-08T08:19:45.086259+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:53:43.166175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099615+0000","last_change":"2023-11-08T08:19:45.094474+0000","last_active":"2023-11-08T08:19:45.099615+0000","last_peered":"2023-11-08T08:19:45.099615+0000","last_clean":"2023-11-08T08:19:45.099615+0000","last_became_active":"2023-11-08T08:19:45.094144+0000","last_became_peered":"2023-11-08T08:19:45.094144+0000","last_unstale":"2023-11-08T08:19:45.099615+0000","last_undegraded":"2023-11-08T08:19:45.099615+0000","last_fullsized":"2023-11-08T08:19:45.099615+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:32.777372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087736+0000","last_change":"2023-11-08T08:19:44.692546+0000","last_active":"2023-11-08T08:19:45.087736+0000","last_peered":"2023-11-08T08:19:45.087736+0000","last_clean":"2023-11-08T08:19:45.087736+0000","last_became_active":"2023-11-08T08:19:44.692320+0000","last_became_peered":"2023-11-08T08:19:44.692320+0000","last_unstale":"2023-11-08T08:19:45.087736+0000","last_undegraded":"2023-11-08T08:19:45.087736+0000","last_fullsized":"2023-11-08T08:19:45.087736+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:38:53.026823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093066+0000","last_change":"2023-11-08T08:19:45.091345+0000","last_active":"2023-11-08T08:19:45.093066+0000","last_peered":"2023-11-08T08:19:45.093066+0000","last_clean":"2023-11-08T08:19:45.093066+0000","last_became_active":"2023-11-08T08:19:45.091173+0000","last_became_peered":"2023-11-08T08:19:45.091173+0000","last_unstale":"2023-11-08T08:19:45.093066+0000","last_undegraded":"2023-11-08T08:19:45.093066+0000","last_fullsized":"2023-11-08T08:19:45.093066+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:19:31.649628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.088305+0000","last_change":"2023-11-08T08:19:44.689843+0000","last_active":"2023-11-08T08:19:45.088305+0000","last_peered":"2023-11-08T08:19:45.088305+0000","last_clean":"2023-11-08T08:19:45.088305+0000","last_became_active":"2023-11-08T08:19:44.689246+0000","last_became_peered":"2023-11-08T08:19:44.689246+0000","last_unstale":"2023-11-08T08:19:45.088305+0000","last_undegraded":"2023-11-08T08:19:45.088305+0000","last_fullsized":"2023-11-08T08:19:45.088305+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:55.598545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094735+0000","last_change":"2023-11-08T08:19:45.085858+0000","last_active":"2023-11-08T08:19:45.094735+0000","last_peered":"2023-11-08T08:19:45.094735+0000","last_clean":"2023-11-08T08:19:45.094735+0000","last_became_active":"2023-11-08T08:19:45.085460+0000","last_became_peered":"2023-11-08T08:19:45.085460+0000","last_unstale":"2023-11-08T08:19:45.094735+0000","last_undegraded":"2023-11-08T08:19:45.094735+0000","last_fullsized":"2023-11-08T08:19:45.094735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:50:48.985661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.072957+0000","last_change":"2023-11-08T08:19:44.691843+0000","last_active":"2023-11-08T08:19:45.072957+0000","last_peered":"2023-11-08T08:19:45.072957+0000","last_clean":"2023-11-08T08:19:45.072957+0000","last_became_active":"2023-11-08T08:19:44.691540+0000","last_became_peered":"2023-11-08T08:19:44.691540+0000","last_unstale":"2023-11-08T08:19:45.072957+0000","last_undegraded":"2023-11-08T08:19:45.072957+0000","last_fullsized":"2023-11-08T08:19:45.072957+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:11:59.871085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092923+0000","last_change":"2023-11-08T08:19:45.091601+0000","last_active":"2023-11-08T08:19:45.092923+0000","last_peered":"2023-11-08T08:19:45.092923+0000","last_clean":"2023-11-08T08:19:45.092923+0000","last_became_active":"2023-11-08T08:19:45.091219+0000","last_became_peered":"2023-11-08T08:19:45.091219+0000","last_unstale":"2023-11-08T08:19:45.092923+0000","last_undegraded":"2023-11-08T08:19:45.092923+0000","last_fullsized":"2023-11-08T08:19:45.092923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:49:26.597569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.077241+0000","last_change":"2023-11-08T08:19:44.690129+0000","last_active":"2023-11-08T08:19:45.077241+0000","last_peered":"2023-11-08T08:19:45.077241+0000","last_clean":"2023-11-08T08:19:45.077241+0000","last_became_active":"2023-11-08T08:19:44.689829+0000","last_became_peered":"2023-11-08T08:19:44.689829+0000","last_unstale":"2023-11-08T08:19:45.077241+0000","last_undegraded":"2023-11-08T08:19:45.077241+0000","last_fullsized":"2023-11-08T08:19:45.077241+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:24:19.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094500+0000","last_change":"2023-11-08T08:19:45.089209+0000","last_active":"2023-11-08T08:19:45.094500+0000","last_peered":"2023-11-08T08:19:45.094500+0000","last_clean":"2023-11-08T08:19:45.094500+0000","last_became_active":"2023-11-08T08:19:45.088913+0000","last_became_peered":"2023-11-08T08:19:45.088913+0000","last_unstale":"2023-11-08T08:19:45.094500+0000","last_undegraded":"2023-11-08T08:19:45.094500+0000","last_fullsized":"2023-11-08T08:19:45.094500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:45:18.850871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.075123+0000","last_change":"2023-11-08T08:19:44.686234+0000","last_active":"2023-11-08T08:19:45.075123+0000","last_peered":"2023-11-08T08:19:45.075123+0000","last_clean":"2023-11-08T08:19:45.075123+0000","last_became_active":"2023-11-08T08:19:44.685928+0000","last_became_peered":"2023-11-08T08:19:44.685928+0000","last_unstale":"2023-11-08T08:19:45.075123+0000","last_undegraded":"2023-11-08T08:19:45.075123+0000","last_fullsized":"2023-11-08T08:19:45.075123+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:56:03.120968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.088736+0000","last_change":"2023-11-08T08:19:45.081453+0000","last_active":"2023-11-08T08:19:45.088736+0000","last_peered":"2023-11-08T08:19:45.088736+0000","last_clean":"2023-11-08T08:19:45.088736+0000","last_became_active":"2023-11-08T08:19:45.081186+0000","last_became_peered":"2023-11-08T08:19:45.081186+0000","last_unstale":"2023-11-08T08:19:45.088736+0000","last_undegraded":"2023-11-08T08:19:45.088736+0000","last_fullsized":"2023-11-08T08:19:45.088736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:56.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085932+0000","last_change":"2023-11-08T08:19:44.691340+0000","last_active":"2023-11-08T08:19:45.085932+0000","last_peered":"2023-11-08T08:19:45.085932+0000","last_clean":"2023-11-08T08:19:45.085932+0000","last_became_active":"2023-11-08T08:19:44.691068+0000","last_became_peered":"2023-11-08T08:19:44.691068+0000","last_unstale":"2023-11-08T08:19:45.085932+0000","last_undegraded":"2023-11-08T08:19:45.085932+0000","last_fullsized":"2023-11-08T08:19:45.085932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:47:41.135797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094589+0000","last_change":"2023-11-08T08:19:45.088539+0000","last_active":"2023-11-08T08:19:45.094589+0000","last_peered":"2023-11-08T08:19:45.094589+0000","last_clean":"2023-11-08T08:19:45.094589+0000","last_became_active":"2023-11-08T08:19:45.087962+0000","last_became_peered":"2023-11-08T08:19:45.087962+0000","last_unstale":"2023-11-08T08:19:45.094589+0000","last_undegraded":"2023-11-08T08:19:45.094589+0000","last_fullsized":"2023-11-08T08:19:45.094589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:08:06.508307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"31'3","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.098027+0000","last_change":"2023-11-08T08:19:44.689537+0000","last_active":"2023-11-08T08:19:45.098027+0000","last_peered":"2023-11-08T08:19:45.098027+0000","last_clean":"2023-11-08T08:19:45.098027+0000","last_became_active":"2023-11-08T08:19:44.689278+0000","last_became_peered":"2023-11-08T08:19:44.689278+0000","last_unstale":"2023-11-08T08:19:45.098027+0000","last_undegraded":"2023-11-08T08:19:45.098027+0000","last_fullsized":"2023-11-08T08:19:45.098027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:49:48.410399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090127+0000","last_change":"2023-11-08T08:19:45.084874+0000","last_active":"2023-11-08T08:19:45.090127+0000","last_peered":"2023-11-08T08:19:45.090127+0000","last_clean":"2023-11-08T08:19:45.090127+0000","last_became_active":"2023-11-08T08:19:45.084603+0000","last_became_peered":"2023-11-08T08:19:45.084603+0000","last_unstale":"2023-11-08T08:19:45.090127+0000","last_undegraded":"2023-11-08T08:19:45.090127+0000","last_fullsized":"2023-11-08T08:19:45.090127+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:06:11.698122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"31'6","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.099967+0000","last_change":"2023-11-08T08:19:44.689549+0000","last_active":"2023-11-08T08:19:45.099967+0000","last_peered":"2023-11-08T08:19:45.099967+0000","last_clean":"2023-11-08T08:19:45.099967+0000","last_became_active":"2023-11-08T08:19:44.689293+0000","last_became_peered":"2023-11-08T08:19:44.689293+0000","last_unstale":"2023-11-08T08:19:45.099967+0000","last_undegraded":"2023-11-08T08:19:45.099967+0000","last_fullsized":"2023-11-08T08:19:45.099967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:35:38.697974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097090+0000","last_change":"2023-11-08T08:19:45.093869+0000","last_active":"2023-11-08T08:19:45.097090+0000","last_peered":"2023-11-08T08:19:45.097090+0000","last_clean":"2023-11-08T08:19:45.097090+0000","last_became_active":"2023-11-08T08:19:45.090749+0000","last_became_peered":"2023-11-08T08:19:45.090749+0000","last_unstale":"2023-11-08T08:19:45.097090+0000","last_undegraded":"2023-11-08T08:19:45.097090+0000","last_fullsized":"2023-11-08T08:19:45.097090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:20:48.892808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093888+0000","last_change":"2023-11-08T08:19:44.693148+0000","last_active":"2023-11-08T08:19:45.093888+0000","last_peered":"2023-11-08T08:19:45.093888+0000","last_clean":"2023-11-08T08:19:45.093888+0000","last_became_active":"2023-11-08T08:19:44.692887+0000","last_became_peered":"2023-11-08T08:19:44.692887+0000","last_unstale":"2023-11-08T08:19:45.093888+0000","last_undegraded":"2023-11-08T08:19:45.093888+0000","last_fullsized":"2023-11-08T08:19:45.093888+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:00.705114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089656+0000","last_change":"2023-11-08T08:19:45.084765+0000","last_active":"2023-11-08T08:19:45.089656+0000","last_peered":"2023-11-08T08:19:45.089656+0000","last_clean":"2023-11-08T08:19:45.089656+0000","last_became_active":"2023-11-08T08:19:45.084449+0000","last_became_peered":"2023-11-08T08:19:45.084449+0000","last_unstale":"2023-11-08T08:19:45.089656+0000","last_undegraded":"2023-11-08T08:19:45.089656+0000","last_fullsized":"2023-11-08T08:19:45.089656+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:12:00.875913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087428+0000","last_change":"2023-11-08T08:19:44.696841+0000","last_active":"2023-11-08T08:19:45.087428+0000","last_peered":"2023-11-08T08:19:45.087428+0000","last_clean":"2023-11-08T08:19:45.087428+0000","last_became_active":"2023-11-08T08:19:44.696606+0000","last_became_peered":"2023-11-08T08:19:44.696606+0000","last_unstale":"2023-11-08T08:19:45.087428+0000","last_undegraded":"2023-11-08T08:19:45.087428+0000","last_fullsized":"2023-11-08T08:19:45.087428+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:54:04.535006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.082650+0000","last_change":"2023-11-08T08:19:45.076678+0000","last_active":"2023-11-08T08:19:45.082650+0000","last_peered":"2023-11-08T08:19:45.082650+0000","last_clean":"2023-11-08T08:19:45.082650+0000","last_became_active":"2023-11-08T08:19:45.076478+0000","last_became_peered":"2023-11-08T08:19:45.076478+0000","last_unstale":"2023-11-08T08:19:45.082650+0000","last_undegraded":"2023-11-08T08:19:45.082650+0000","last_fullsized":"2023-11-08T08:19:45.082650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:49:26.597569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085707+0000","last_change":"2023-11-08T08:19:44.692380+0000","last_active":"2023-11-08T08:19:45.085707+0000","last_peered":"2023-11-08T08:19:45.085707+0000","last_clean":"2023-11-08T08:19:45.085707+0000","last_became_active":"2023-11-08T08:19:44.690445+0000","last_became_peered":"2023-11-08T08:19:44.690445+0000","last_unstale":"2023-11-08T08:19:45.085707+0000","last_undegraded":"2023-11-08T08:19:45.085707+0000","last_fullsized":"2023-11-08T08:19:45.085707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:12.828909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093091+0000","last_change":"2023-11-08T08:19:45.087566+0000","last_active":"2023-11-08T08:19:45.093091+0000","last_peered":"2023-11-08T08:19:45.093091+0000","last_clean":"2023-11-08T08:19:45.093091+0000","last_became_active":"2023-11-08T08:19:45.087288+0000","last_became_peered":"2023-11-08T08:19:45.087288+0000","last_unstale":"2023-11-08T08:19:45.093091+0000","last_undegraded":"2023-11-08T08:19:45.093091+0000","last_fullsized":"2023-11-08T08:19:45.093091+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:47.846149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"31'8","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090368+0000","last_change":"2023-11-08T08:19:44.690997+0000","last_active":"2023-11-08T08:19:45.090368+0000","last_peered":"2023-11-08T08:19:45.090368+0000","last_clean":"2023-11-08T08:19:45.090368+0000","last_became_active":"2023-11-08T08:19:44.690731+0000","last_became_peered":"2023-11-08T08:19:44.690731+0000","last_unstale":"2023-11-08T08:19:45.090368+0000","last_undegraded":"2023-11-08T08:19:45.090368+0000","last_fullsized":"2023-11-08T08:19:45.090368+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:04:54.456244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.089074+0000","last_change":"2023-11-08T08:19:45.084220+0000","last_active":"2023-11-08T08:19:45.089074+0000","last_peered":"2023-11-08T08:19:45.089074+0000","last_clean":"2023-11-08T08:19:45.089074+0000","last_became_active":"2023-11-08T08:19:45.083895+0000","last_became_peered":"2023-11-08T08:19:45.083895+0000","last_unstale":"2023-11-08T08:19:45.089074+0000","last_undegraded":"2023-11-08T08:19:45.089074+0000","last_fullsized":"2023-11-08T08:19:45.089074+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:11.342423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.084240+0000","last_change":"2023-11-08T08:19:44.688634+0000","last_active":"2023-11-08T08:19:45.084240+0000","last_peered":"2023-11-08T08:19:45.084240+0000","last_clean":"2023-11-08T08:19:45.084240+0000","last_became_active":"2023-11-08T08:19:44.688382+0000","last_became_peered":"2023-11-08T08:19:44.688382+0000","last_unstale":"2023-11-08T08:19:45.084240+0000","last_undegraded":"2023-11-08T08:19:45.084240+0000","last_fullsized":"2023-11-08T08:19:45.084240+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:06:16.804906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092344+0000","last_change":"2023-11-08T08:19:45.088639+0000","last_active":"2023-11-08T08:19:45.092344+0000","last_peered":"2023-11-08T08:19:45.092344+0000","last_clean":"2023-11-08T08:19:45.092344+0000","last_became_active":"2023-11-08T08:19:45.088394+0000","last_became_peered":"2023-11-08T08:19:45.088394+0000","last_unstale":"2023-11-08T08:19:45.092344+0000","last_undegraded":"2023-11-08T08:19:45.092344+0000","last_fullsized":"2023-11-08T08:19:45.092344+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:01.709942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.097998+0000","last_change":"2023-11-08T08:19:44.690147+0000","last_active":"2023-11-08T08:19:45.097998+0000","last_peered":"2023-11-08T08:19:45.097998+0000","last_clean":"2023-11-08T08:19:45.097998+0000","last_became_active":"2023-11-08T08:19:44.689868+0000","last_became_peered":"2023-11-08T08:19:44.689868+0000","last_unstale":"2023-11-08T08:19:45.097998+0000","last_undegraded":"2023-11-08T08:19:45.097998+0000","last_fullsized":"2023-11-08T08:19:45.097998+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:01:11.769036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090226+0000","last_change":"2023-11-08T08:19:45.086841+0000","last_active":"2023-11-08T08:19:45.090226+0000","last_peered":"2023-11-08T08:19:45.090226+0000","last_clean":"2023-11-08T08:19:45.090226+0000","last_became_active":"2023-11-08T08:19:45.086558+0000","last_became_peered":"2023-11-08T08:19:45.086558+0000","last_unstale":"2023-11-08T08:19:45.090226+0000","last_undegraded":"2023-11-08T08:19:45.090226+0000","last_fullsized":"2023-11-08T08:19:45.090226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:03:27.481450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090742+0000","last_change":"2023-11-08T08:19:44.689939+0000","last_active":"2023-11-08T08:19:45.090742+0000","last_peered":"2023-11-08T08:19:45.090742+0000","last_clean":"2023-11-08T08:19:45.090742+0000","last_became_active":"2023-11-08T08:19:44.689669+0000","last_became_peered":"2023-11-08T08:19:44.689669+0000","last_unstale":"2023-11-08T08:19:45.090742+0000","last_undegraded":"2023-11-08T08:19:45.090742+0000","last_fullsized":"2023-11-08T08:19:45.090742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:46:22.351765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.094400+0000","last_change":"2023-11-08T08:19:45.090531+0000","last_active":"2023-11-08T08:19:45.094400+0000","last_peered":"2023-11-08T08:19:45.094400+0000","last_clean":"2023-11-08T08:19:45.094400+0000","last_became_active":"2023-11-08T08:19:45.090181+0000","last_became_peered":"2023-11-08T08:19:45.090181+0000","last_unstale":"2023-11-08T08:19:45.094400+0000","last_undegraded":"2023-11-08T08:19:45.094400+0000","last_fullsized":"2023-11-08T08:19:45.094400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:47.846149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.083212+0000","last_change":"2023-11-08T08:19:44.686547+0000","last_active":"2023-11-08T08:19:45.083212+0000","last_peered":"2023-11-08T08:19:45.083212+0000","last_clean":"2023-11-08T08:19:45.083212+0000","last_became_active":"2023-11-08T08:19:44.686251+0000","last_became_peered":"2023-11-08T08:19:44.686251+0000","last_unstale":"2023-11-08T08:19:45.083212+0000","last_undegraded":"2023-11-08T08:19:45.083212+0000","last_fullsized":"2023-11-08T08:19:45.083212+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:04:54.456244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095251+0000","last_change":"2023-11-08T08:19:45.088356+0000","last_active":"2023-11-08T08:19:45.095251+0000","last_peered":"2023-11-08T08:19:45.095251+0000","last_clean":"2023-11-08T08:19:45.095251+0000","last_became_active":"2023-11-08T08:19:45.087954+0000","last_became_peered":"2023-11-08T08:19:45.087954+0000","last_unstale":"2023-11-08T08:19:45.095251+0000","last_undegraded":"2023-11-08T08:19:45.095251+0000","last_fullsized":"2023-11-08T08:19:45.095251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:50:17.804798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093492+0000","last_change":"2023-11-08T08:19:44.686890+0000","last_active":"2023-11-08T08:19:45.093492+0000","last_peered":"2023-11-08T08:19:45.093492+0000","last_clean":"2023-11-08T08:19:45.093492+0000","last_became_active":"2023-11-08T08:19:44.686592+0000","last_became_peered":"2023-11-08T08:19:44.686592+0000","last_unstale":"2023-11-08T08:19:45.093492+0000","last_undegraded":"2023-11-08T08:19:45.093492+0000","last_fullsized":"2023-11-08T08:19:45.093492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:38:53.026823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093613+0000","last_change":"2023-11-08T08:19:45.087149+0000","last_active":"2023-11-08T08:19:45.093613+0000","last_peered":"2023-11-08T08:19:45.093613+0000","last_clean":"2023-11-08T08:19:45.093613+0000","last_became_active":"2023-11-08T08:19:45.086409+0000","last_became_peered":"2023-11-08T08:19:45.086409+0000","last_unstale":"2023-11-08T08:19:45.093613+0000","last_undegraded":"2023-11-08T08:19:45.093613+0000","last_fullsized":"2023-11-08T08:19:45.093613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:19:18.354270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087524+0000","last_change":"2023-11-08T08:19:44.694936+0000","last_active":"2023-11-08T08:19:45.087524+0000","last_peered":"2023-11-08T08:19:45.087524+0000","last_clean":"2023-11-08T08:19:45.087524+0000","last_became_active":"2023-11-08T08:19:44.694644+0000","last_became_peered":"2023-11-08T08:19:44.694644+0000","last_unstale":"2023-11-08T08:19:45.087524+0000","last_undegraded":"2023-11-08T08:19:45.087524+0000","last_fullsized":"2023-11-08T08:19:45.087524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:08:50.314631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.082986+0000","last_change":"2023-11-08T08:19:45.080947+0000","last_active":"2023-11-08T08:19:45.082986+0000","last_peered":"2023-11-08T08:19:45.082986+0000","last_clean":"2023-11-08T08:19:45.082986+0000","last_became_active":"2023-11-08T08:19:45.080646+0000","last_became_peered":"2023-11-08T08:19:45.080646+0000","last_unstale":"2023-11-08T08:19:45.082986+0000","last_undegraded":"2023-11-08T08:19:45.082986+0000","last_fullsized":"2023-11-08T08:19:45.082986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:12:00.875913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.076750+0000","last_change":"2023-11-08T08:19:44.691404+0000","last_active":"2023-11-08T08:19:45.076750+0000","last_peered":"2023-11-08T08:19:45.076750+0000","last_clean":"2023-11-08T08:19:45.076750+0000","last_became_active":"2023-11-08T08:19:44.691130+0000","last_became_peered":"2023-11-08T08:19:44.691130+0000","last_unstale":"2023-11-08T08:19:45.076750+0000","last_undegraded":"2023-11-08T08:19:45.076750+0000","last_fullsized":"2023-11-08T08:19:45.076750+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:49:46.841322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.093698+0000","last_change":"2023-11-08T08:19:45.087499+0000","last_active":"2023-11-08T08:19:45.093698+0000","last_peered":"2023-11-08T08:19:45.093698+0000","last_clean":"2023-11-08T08:19:45.093698+0000","last_became_active":"2023-11-08T08:19:45.087229+0000","last_became_peered":"2023-11-08T08:19:45.087229+0000","last_unstale":"2023-11-08T08:19:45.093698+0000","last_undegraded":"2023-11-08T08:19:45.093698+0000","last_fullsized":"2023-11-08T08:19:45.093698+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:20.841622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.084758+0000","last_change":"2023-11-08T08:19:44.693768+0000","last_active":"2023-11-08T08:19:45.084758+0000","last_peered":"2023-11-08T08:19:45.084758+0000","last_clean":"2023-11-08T08:19:45.084758+0000","last_became_active":"2023-11-08T08:19:44.693483+0000","last_became_peered":"2023-11-08T08:19:44.693483+0000","last_unstale":"2023-11-08T08:19:45.084758+0000","last_undegraded":"2023-11-08T08:19:45.084758+0000","last_fullsized":"2023-11-08T08:19:45.084758+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:24:39.933425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095318+0000","last_change":"2023-11-08T08:19:45.089276+0000","last_active":"2023-11-08T08:19:45.095318+0000","last_peered":"2023-11-08T08:19:45.095318+0000","last_clean":"2023-11-08T08:19:45.095318+0000","last_became_active":"2023-11-08T08:19:45.089005+0000","last_became_peered":"2023-11-08T08:19:45.089005+0000","last_unstale":"2023-11-08T08:19:45.095318+0000","last_undegraded":"2023-11-08T08:19:45.095318+0000","last_fullsized":"2023-11-08T08:19:45.095318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:01:43.871569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.085788+0000","last_change":"2023-11-08T08:19:44.690469+0000","last_active":"2023-11-08T08:19:45.085788+0000","last_peered":"2023-11-08T08:19:45.085788+0000","last_clean":"2023-11-08T08:19:45.085788+0000","last_became_active":"2023-11-08T08:19:44.690194+0000","last_became_peered":"2023-11-08T08:19:44.690194+0000","last_unstale":"2023-11-08T08:19:45.085788+0000","last_undegraded":"2023-11-08T08:19:45.085788+0000","last_fullsized":"2023-11-08T08:19:45.085788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:57:00.705114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.684383+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:19:44.684383+0000","last_peered":"2023-11-08T08:19:44.684383+0000","last_clean":"2023-11-08T08:19:44.684383+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:19:44.684383+0000","last_undegraded":"2023-11-08T08:19:44.684383+0000","last_fullsized":"2023-11-08T08:19:44.684383+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092851+0000","last_change":"2023-11-08T08:19:45.090352+0000","last_active":"2023-11-08T08:19:45.092851+0000","last_peered":"2023-11-08T08:19:45.092851+0000","last_clean":"2023-11-08T08:19:45.092851+0000","last_became_active":"2023-11-08T08:19:45.090100+0000","last_became_peered":"2023-11-08T08:19:45.090100+0000","last_unstale":"2023-11-08T08:19:45.092851+0000","last_undegraded":"2023-11-08T08:19:45.092851+0000","last_fullsized":"2023-11-08T08:19:45.092851+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:04:55.461072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.074440+0000","last_change":"2023-11-08T08:19:44.692860+0000","last_active":"2023-11-08T08:19:45.074440+0000","last_peered":"2023-11-08T08:19:45.074440+0000","last_clean":"2023-11-08T08:19:45.074440+0000","last_became_active":"2023-11-08T08:19:44.692585+0000","last_became_peered":"2023-11-08T08:19:44.692585+0000","last_unstale":"2023-11-08T08:19:45.074440+0000","last_undegraded":"2023-11-08T08:19:45.074440+0000","last_fullsized":"2023-11-08T08:19:45.074440+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:10.337595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.679995+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:19:44.679995+0000","last_peered":"2023-11-08T08:19:44.679995+0000","last_clean":"2023-11-08T08:19:44.679995+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:19:44.679995+0000","last_undegraded":"2023-11-08T08:19:44.679995+0000","last_fullsized":"2023-11-08T08:19:44.679995+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.083521+0000","last_change":"2023-11-08T08:19:45.081447+0000","last_active":"2023-11-08T08:19:45.083521+0000","last_peered":"2023-11-08T08:19:45.083521+0000","last_clean":"2023-11-08T08:19:45.083521+0000","last_became_active":"2023-11-08T08:19:45.081128+0000","last_became_peered":"2023-11-08T08:19:45.081128+0000","last_unstale":"2023-11-08T08:19:45.083521+0000","last_undegraded":"2023-11-08T08:19:45.083521+0000","last_fullsized":"2023-11-08T08:19:45.083521+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:06:17.809734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"31'2","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095573+0000","last_change":"2023-11-08T08:19:44.689594+0000","last_active":"2023-11-08T08:19:45.095573+0000","last_peered":"2023-11-08T08:19:45.095573+0000","last_clean":"2023-11-08T08:19:45.095573+0000","last_became_active":"2023-11-08T08:19:44.689351+0000","last_became_peered":"2023-11-08T08:19:44.689351+0000","last_unstale":"2023-11-08T08:19:45.095573+0000","last_undegraded":"2023-11-08T08:19:45.095573+0000","last_fullsized":"2023-11-08T08:19:45.095573+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:06:10.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.677930+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:19:44.677930+0000","last_peered":"2023-11-08T08:19:44.677930+0000","last_clean":"2023-11-08T08:19:44.677930+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:19:44.677930+0000","last_undegraded":"2023-11-08T08:19:44.677930+0000","last_fullsized":"2023-11-08T08:19:44.677930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":174,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.682134+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:19:44.682134+0000","last_peered":"2023-11-08T08:19:44.682134+0000","last_clean":"2023-11-08T08:19:44.682134+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:19:44.682134+0000","last_undegraded":"2023-11-08T08:19:44.682134+0000","last_fullsized":"2023-11-08T08:19:44.682134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.090347+0000","last_change":"2023-11-08T08:19:45.083642+0000","last_active":"2023-11-08T08:19:45.090347+0000","last_peered":"2023-11-08T08:19:45.090347+0000","last_clean":"2023-11-08T08:19:45.090347+0000","last_became_active":"2023-11-08T08:19:45.083392+0000","last_became_peered":"2023-11-08T08:19:45.083392+0000","last_unstale":"2023-11-08T08:19:45.090347+0000","last_undegraded":"2023-11-08T08:19:45.090347+0000","last_fullsized":"2023-11-08T08:19:45.090347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:23:40.198997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.073647+0000","last_change":"2023-11-08T08:19:44.689997+0000","last_active":"2023-11-08T08:19:45.073647+0000","last_peered":"2023-11-08T08:19:45.073647+0000","last_clean":"2023-11-08T08:19:45.073647+0000","last_became_active":"2023-11-08T08:19:44.689657+0000","last_became_peered":"2023-11-08T08:19:44.689657+0000","last_unstale":"2023-11-08T08:19:45.073647+0000","last_undegraded":"2023-11-08T08:19:45.073647+0000","last_fullsized":"2023-11-08T08:19:45.073647+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:18:04.463027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.681188+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:19:44.681188+0000","last_peered":"2023-11-08T08:19:44.681188+0000","last_clean":"2023-11-08T08:19:44.681188+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:19:44.681188+0000","last_undegraded":"2023-11-08T08:19:44.681188+0000","last_fullsized":"2023-11-08T08:19:44.681188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.095611+0000","last_change":"2023-11-08T08:19:45.092553+0000","last_active":"2023-11-08T08:19:45.095611+0000","last_peered":"2023-11-08T08:19:45.095611+0000","last_clean":"2023-11-08T08:19:45.095611+0000","last_became_active":"2023-11-08T08:19:45.092280+0000","last_became_peered":"2023-11-08T08:19:45.092280+0000","last_unstale":"2023-11-08T08:19:45.095611+0000","last_undegraded":"2023-11-08T08:19:45.095611+0000","last_fullsized":"2023-11-08T08:19:45.095611+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:08:06.508307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.069762+0000","last_change":"2023-11-08T08:19:44.688419+0000","last_active":"2023-11-08T08:19:45.069762+0000","last_peered":"2023-11-08T08:19:45.069762+0000","last_clean":"2023-11-08T08:19:45.069762+0000","last_became_active":"2023-11-08T08:19:44.688133+0000","last_became_peered":"2023-11-08T08:19:44.688133+0000","last_unstale":"2023-11-08T08:19:45.069762+0000","last_undegraded":"2023-11-08T08:19:45.069762+0000","last_fullsized":"2023-11-08T08:19:45.069762+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:54:04.535006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.684666+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:19:44.684666+0000","last_peered":"2023-11-08T08:19:44.684666+0000","last_clean":"2023-11-08T08:19:44.684666+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:19:44.684666+0000","last_undegraded":"2023-11-08T08:19:44.684666+0000","last_fullsized":"2023-11-08T08:19:44.684666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.102590+0000","last_change":"2023-11-08T08:19:45.098102+0000","last_active":"2023-11-08T08:19:45.102590+0000","last_peered":"2023-11-08T08:19:45.102590+0000","last_clean":"2023-11-08T08:19:45.102590+0000","last_became_active":"2023-11-08T08:19:45.097869+0000","last_became_peered":"2023-11-08T08:19:45.097869+0000","last_unstale":"2023-11-08T08:19:45.102590+0000","last_undegraded":"2023-11-08T08:19:45.102590+0000","last_fullsized":"2023-11-08T08:19:45.102590+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:29:56.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.073605+0000","last_change":"2023-11-08T08:19:44.684683+0000","last_active":"2023-11-08T08:19:45.073605+0000","last_peered":"2023-11-08T08:19:45.073605+0000","last_clean":"2023-11-08T08:19:45.073605+0000","last_became_active":"2023-11-08T08:19:44.684132+0000","last_became_peered":"2023-11-08T08:19:44.684132+0000","last_unstale":"2023-11-08T08:19:45.073605+0000","last_undegraded":"2023-11-08T08:19:45.073605+0000","last_fullsized":"2023-11-08T08:19:45.073605+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:53:10.877639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.682243+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:19:44.682243+0000","last_peered":"2023-11-08T08:19:44.682243+0000","last_clean":"2023-11-08T08:19:44.682243+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:19:44.682243+0000","last_undegraded":"2023-11-08T08:19:44.682243+0000","last_fullsized":"2023-11-08T08:19:44.682243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.092944+0000","last_change":"2023-11-08T08:19:45.091802+0000","last_active":"2023-11-08T08:19:45.092944+0000","last_peered":"2023-11-08T08:19:45.092944+0000","last_clean":"2023-11-08T08:19:45.092944+0000","last_became_active":"2023-11-08T08:19:45.091517+0000","last_became_peered":"2023-11-08T08:19:45.091517+0000","last_unstale":"2023-11-08T08:19:45.092944+0000","last_undegraded":"2023-11-08T08:19:45.092944+0000","last_fullsized":"2023-11-08T08:19:45.092944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:13.833737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.069422+0000","last_change":"2023-11-08T08:19:44.692578+0000","last_active":"2023-11-08T08:19:45.069422+0000","last_peered":"2023-11-08T08:19:45.069422+0000","last_clean":"2023-11-08T08:19:45.069422+0000","last_became_active":"2023-11-08T08:19:44.692276+0000","last_became_peered":"2023-11-08T08:19:44.692276+0000","last_unstale":"2023-11-08T08:19:45.069422+0000","last_undegraded":"2023-11-08T08:19:45.069422+0000","last_fullsized":"2023-11-08T08:19:45.069422+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:08:50.314631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.687063+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:19:44.687063+0000","last_peered":"2023-11-08T08:19:44.687063+0000","last_clean":"2023-11-08T08:19:44.687063+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:19:44.687063+0000","last_undegraded":"2023-11-08T08:19:44.687063+0000","last_fullsized":"2023-11-08T08:19:44.687063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.087356+0000","last_change":"2023-11-08T08:19:45.084782+0000","last_active":"2023-11-08T08:19:45.087356+0000","last_peered":"2023-11-08T08:19:45.087356+0000","last_clean":"2023-11-08T08:19:45.087356+0000","last_became_active":"2023-11-08T08:19:45.084512+0000","last_became_peered":"2023-11-08T08:19:45.084512+0000","last_unstale":"2023-11-08T08:19:45.087356+0000","last_undegraded":"2023-11-08T08:19:45.087356+0000","last_fullsized":"2023-11-08T08:19:45.087356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:44.667604+0000","last_clean_scrub_stamp":"2023-11-08T08:19:44.667604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:38:54.031651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-08T08:19:45.074606+0000","last_change":"2023-11-08T08:19:44.693135+0000","last_active":"2023-11-08T08:19:45.074606+0000","last_peered":"2023-11-08T08:19:45.074606+0000","last_clean":"2023-11-08T08:19:45.074606+0000","last_became_active":"2023-11-08T08:19:44.692681+0000","last_became_peered":"2023-11-08T08:19:44.692681+0000","last_unstale":"2023-11-08T08:19:45.074606+0000","last_undegraded":"2023-11-08T08:19:45.074606+0000","last_fullsized":"2023-11-08T08:19:45.074606+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:19:43.662776+0000","last_clean_scrub_stamp":"2023-11-08T08:19:43.662776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:43:12.828909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-08T08:19:44.686519+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:19:44.686519+0000","last_peered":"2023-11-08T08:19:44.686519+0000","last_clean":"2023-11-08T08:19:44.686519+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:19:44.686519+0000","last_undegraded":"2023-11-08T08:19:44.686519+0000","last_fullsized":"2023-11-08T08:19:44.686519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8224,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363616,"statfs":{"total":96636764160,"available":96628342784,"internally_reserved":0,"allocated":352256,"data_stored":140604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.59699999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":5.8410000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.623}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.433}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51800000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8268,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363572,"statfs":{"total":96636764160,"available":96628297728,"internally_reserved":0,"allocated":331776,"data_stored":132145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45500000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.58499999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.65900000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8196,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363644,"statfs":{"total":96636764160,"available":96628371456,"internally_reserved":0,"allocated":323584,"data_stored":126778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.246},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.32900000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55900000000000005}]},{"osd":6,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55100000000000005}]},{"osd":7,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.48799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8356,"kb_used_data":348,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363484,"statfs":{"total":96636764160,"available":96628207616,"internally_reserved":0,"allocated":356352,"data_stored":140626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.34399999999999997}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.49399999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.46100000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8364,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363476,"statfs":{"total":96636764160,"available":96628199424,"internally_reserved":0,"allocated":364544,"data_stored":142648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.19},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.31900000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.307},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"last":0.245}]},{"osd":4,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.495}]},{"osd":5,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51000000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.58099999999999996}]},{"osd":7,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9508,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362332,"statfs":{"total":96636764160,"available":96627027968,"internally_reserved":0,"allocated":946176,"data_stored":730992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.247},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.372}]},{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.5070000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.313}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.28699999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.69999999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.68799999999999994}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.80100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8432,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363408,"statfs":{"total":96636764160,"available":96628129792,"internally_reserved":0,"allocated":368640,"data_stored":142206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.5}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.58399999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40500000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.38200000000000001}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9444,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94362396,"statfs":{"total":96636764160,"available":96627093504,"internally_reserved":0,"allocated":946176,"data_stored":731468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.6790000000000003,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.7270000000000003,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.45100000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.501},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":7.6079999999999997,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.0209999999999999,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.373}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":9.7550000000000008,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.5489999999999999,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.52400000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.35999999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:19:49.508 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-08T08:19:49.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-11-08T08:19:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.669+0000 7fd8e4491700 1 Processor -- start 2023-11-08T08:19:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.669+0000 7fd8e4491700 1 -- start start 2023-11-08T08:19:49.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.670+0000 7fd8e4491700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc1599b0 0x7fd8dc15d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.670+0000 7fd8e4491700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc159ec0 con 0x7fd8dc0af0a0 2023-11-08T08:19:49.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.670+0000 7fd8e4491700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc15aa60 con 0x7fd8dc0b1650 2023-11-08T08:19:49.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.670+0000 7fd8e4491700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc15d8a0 con 0x7fd8dc1599b0 2023-11-08T08:19:49.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.670+0000 7fd8e2a2e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd8dc0af0a0 0x7fd8dc15a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36568/0 (socket says 172.21.15.116:36568) 2023-11-08T08:19:49.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.670+0000 7fd8e2a2e700 1 -- 172.21.15.116:0/809050026 learned_addr learned my addr 172.21.15.116:0/809050026 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:49.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086126310 0 0) 0x7fd8dc159ec0 con 0x7fd8dc0af0a0 2023-11-08T08:19:49.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c80035b0 con 0x7fd8dc0af0a0 2023-11-08T08:19:49.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e1a2c700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc1599b0 0x7fd8dc15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2763869044 0 0) 0x7fd8c80035b0 con 0x7fd8dc0af0a0 2023-11-08T08:19:49.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc1599b0 msgr2=0x7fd8dc15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:49.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc1599b0 0x7fd8dc15d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 >> v1:172.21.15.129:6789/0 conn(0x7fd8dc0b1650 legacy=0x7fd8dc0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc15db80 con 0x7fd8dc0af0a0 2023-11-08T08:19:49.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fd8d40019f0 con 0x7fd8dc0af0a0 2023-11-08T08:19:49.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 >> v1:172.21.15.116:6789/0 conn(0x7fd8dc0af0a0 legacy=0x7fd8dc15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8c8003a30 0x7fd8c8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8004850 0x7fd8c8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8c8005200 0x7fd8c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc15d8a0 con 0x7fd8c8003a30 2023-11-08T08:19:49.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c800bf50 con 0x7fd8c8004850 2023-11-08T08:19:49.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c800e2a0 con 0x7fd8c8005200 2023-11-08T08:19:49.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8c8003a30 0x7fd8c8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e1a2c700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8004850 0x7fd8c8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e2a2e700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8c8005200 0x7fd8c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 -- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8c8005200 msgr2=0x7fd8c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:49.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8c8005200 0x7fd8c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 -- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8004850 msgr2=0x7fd8c8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:49.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8004850 0x7fd8c8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 -- 172.21.15.116:0/809050026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c80035b0 con 0x7fd8c8003a30 2023-11-08T08:19:49.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e2a2e700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8c8005200 0x7fd8c800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:49.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e1a2c700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8004850 0x7fd8c8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:49.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e222d700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8c8003a30 0x7fd8c8003e40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc001830 tx=0x7fd8c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6bead11c5bd449 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:49.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.671+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc0012f0 con 0x7fd8c8003a30 2023-11-08T08:19:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc0052f0 con 0x7fd8c8003a30 2023-11-08T08:19:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e122b700 1 -- 172.21.15.116:0/809050026 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc006860 con 0x7fd8c8003a30 2023-11-08T08:19:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e4491700 1 -- 172.21.15.116:0/809050026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8c8003a30 msgr2=0x7fd8c8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e4491700 1 --2- 172.21.15.116:0/809050026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8c8003a30 0x7fd8c8003e40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc001830 tx=0x7fd8c8005880 comp rx=0 tx=0).stop 2023-11-08T08:19:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e2a2e700 1 -- 172.21.15.116:0/809050026 reap_dead start 2023-11-08T08:19:49.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e4491700 1 -- 172.21.15.116:0/809050026 shutdown_connections 2023-11-08T08:19:49.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e4491700 1 -- 172.21.15.116:0/809050026 >> 172.21.15.116:0/809050026 conn(0x7fd8dc06c010 msgr2=0x7fd8dc0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:49.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e4491700 1 -- 172.21.15.116:0/809050026 shutdown_connections 2023-11-08T08:19:49.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.672+0000 7fd8e4491700 1 -- 172.21.15.116:0/809050026 wait complete. 2023-11-08T08:19:49.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.673+0000 7fd8e4491700 1 Processor -- start 2023-11-08T08:19:49.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.673+0000 7fd8e4491700 1 -- start start 2023-11-08T08:19:49.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e4491700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8003a30 0x7fd8dc0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e4491700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc157db0 0x7fd8dc1512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e4491700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 0x7fd8dc0b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e4491700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc158220 con 0x7fd8dc153320 2023-11-08T08:19:49.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e4491700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc1522f0 con 0x7fd8c8003a30 2023-11-08T08:19:49.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e4491700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0b4650 con 0x7fd8dc157db0 2023-11-08T08:19:49.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 0x7fd8dc0b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 0x7fd8dc0b0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60262/0 (socket says 172.21.15.116:60262) 2023-11-08T08:19:49.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 -- 172.21.15.116:0/73010322 learned_addr learned my addr 172.21.15.116:0/73010322 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:49.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e222d700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8003a30 0x7fd8dc0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e1a2c700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc157db0 0x7fd8dc1512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 -- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc157db0 msgr2=0x7fd8dc1512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc157db0 0x7fd8dc1512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 -- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8003a30 msgr2=0x7fd8dc0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8003a30 0x7fd8dc0b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 -- 172.21.15.116:0/73010322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc15d050 con 0x7fd8dc153320 2023-11-08T08:19:49.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e1a2c700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc157db0 0x7fd8dc1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:49.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.674+0000 7fd8e2a2e700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 0x7fd8dc0b0800 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4002010 tx=0x7fd8d4007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:49.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.675+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400ed70 con 0x7fd8dc153320 2023-11-08T08:19:49.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.675+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d400eed0 con 0x7fd8dc153320 2023-11-08T08:19:49.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.675+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc15de60 con 0x7fd8dc153320 2023-11-08T08:19:49.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.675+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4013620 con 0x7fd8dc153320 2023-11-08T08:19:49.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.675+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc15d1e0 con 0x7fd8dc153320 2023-11-08T08:19:49.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.675+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc0743f0 con 0x7fd8dc153320 2023-11-08T08:19:49.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.676+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd8d4013780 con 0x7fd8dc153320 2023-11-08T08:19:49.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.676+0000 7fd8d2ffd700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8c00682f0 0x7fd8c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:49.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.677+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd8d4086e60 con 0x7fd8dc153320 2023-11-08T08:19:49.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.677+0000 7fd8e222d700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8c00682f0 0x7fd8c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:49.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.677+0000 7fd8e222d700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8c00682f0 0x7fd8c006a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc014040 tx=0x7fd8cc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:49.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.680+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8d40b25e0 con 0x7fd8dc153320 2023-11-08T08:19:49.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.934+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fd8dc06e620 con 0x7fd8dc153320 2023-11-08T08:19:49.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.937+0000 7fd8d2ffd700 1 -- 172.21.15.116:0/73010322 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd8d4056f60 con 0x7fd8dc153320 2023-11-08T08:19:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8c00682f0 msgr2=0x7fd8c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8c00682f0 0x7fd8c006a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc014040 tx=0x7fd8cc009030 comp rx=0 tx=0).stop 2023-11-08T08:19:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 msgr2=0x7fd8dc0b0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 0x7fd8dc0b0800 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4002010 tx=0x7fd8d4007c90 comp rx=0 tx=0).stop 2023-11-08T08:19:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 shutdown_connections 2023-11-08T08:19:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8dc153320 0x7fd8dc0b0800 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8c00682f0 0x7fd8c006a7a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c8003a30 0x7fd8dc0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 --2- 172.21.15.116:0/73010322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8dc157db0 0x7fd8dc1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 >> 172.21.15.116:0/73010322 conn(0x7fd8dc06c010 msgr2=0x7fd8dc15a250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 shutdown_connections 2023-11-08T08:19:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:49.940+0000 7fd8e4491700 1 -- 172.21.15.116:0/73010322 wait complete. 2023-11-08T08:19:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.0 2023-11-08T08:19:49.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:19:50.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.072+0000 7f0693d5e700 1 Processor -- start 2023-11-08T08:19:50.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.072+0000 7f0693d5e700 1 -- start start 2023-11-08T08:19:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0693d5e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f068c15e010 0x7f068c0ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0693d5e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c15af30 con 0x7f068c0d15f0 2023-11-08T08:19:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0693d5e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c15ef40 con 0x7f068c06e400 2023-11-08T08:19:50.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0693d5e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068c15a9d0 con 0x7f068c15e010 2023-11-08T08:19:50.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f06912f9700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f068c0d15f0 0x7f068c15b3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36570/0 (socket says 172.21.15.116:36570) 2023-11-08T08:19:50.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f06912f9700 1 -- 172.21.15.116:0/1933833212 learned_addr learned my addr 172.21.15.116:0/1933833212 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:50.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f06922fb700 1 --2- 172.21.15.116:0/1933833212 >> v2:172.21.15.129:3300/0 conn(0x7f068c15e010 0x7f068c0ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753425868 0 0) 0x7f068c15af30 con 0x7f068c0d15f0 2023-11-08T08:19:50.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780035b0 con 0x7f068c0d15f0 2023-11-08T08:19:50.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172196352 0 0) 0x7f06780035b0 con 0x7f068c0d15f0 2023-11-08T08:19:50.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 >> v2:172.21.15.129:3300/0 conn(0x7f068c15e010 msgr2=0x7f068c0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 --2- 172.21.15.116:0/1933833212 >> v2:172.21.15.129:3300/0 conn(0x7f068c15e010 0x7f068c0ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 >> v1:172.21.15.129:6789/0 conn(0x7f068c06e400 legacy=0x7f068c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c15e980 con 0x7f068c0d15f0 2023-11-08T08:19:50.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f067c001670 con 0x7f068c0d15f0 2023-11-08T08:19:50.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 >> v1:172.21.15.116:6789/0 conn(0x7f068c0d15f0 legacy=0x7f068c15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0678003a00 0x7f0678003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06780043f0 0x7f06780048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f06780050b0 0x7f067800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067800c000 con 0x7f06780043f0 2023-11-08T08:19:50.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067800c140 con 0x7f0678003a00 2023-11-08T08:19:50.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067800c300 con 0x7f06780050b0 2023-11-08T08:19:50.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.073+0000 7f06912f9700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06780043f0 0x7f06780048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0691afa700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0678003a00 0x7f0678003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06922fb700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f06780050b0 0x7f067800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06912f9700 1 -- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f06780050b0 msgr2=0x7f067800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06912f9700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f06780050b0 0x7f067800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06912f9700 1 -- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0678003a00 msgr2=0x7f0678003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06912f9700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0678003a00 0x7f0678003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06912f9700 1 -- 172.21.15.116:0/1933833212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780035b0 con 0x7f06780043f0 2023-11-08T08:19:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06922fb700 1 -- 172.21.15.116:0/1933833212 reap_dead start 2023-11-08T08:19:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f06912f9700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06780043f0 0x7f06780048b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f068c0d1a00 tx=0x7f0678005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90120cf261da34ec server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00cd80 con 0x7f06780043f0 2023-11-08T08:19:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c005960 con 0x7f06780043f0 2023-11-08T08:19:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0690af8700 1 -- 172.21.15.116:0/1933833212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c005d10 con 0x7f06780043f0 2023-11-08T08:19:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0693d5e700 1 -- 172.21.15.116:0/1933833212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06780043f0 msgr2=0x7f06780048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0693d5e700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06780043f0 0x7f06780048b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f068c0d1a00 tx=0x7f0678005880 comp rx=0 tx=0).stop 2023-11-08T08:19:50.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0693d5e700 1 -- 172.21.15.116:0/1933833212 shutdown_connections 2023-11-08T08:19:50.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0693d5e700 1 --2- 172.21.15.116:0/1933833212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06780043f0 0x7f06780048b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.074+0000 7f0693d5e700 1 -- 172.21.15.116:0/1933833212 >> 172.21.15.116:0/1933833212 conn(0x7f068c06a270 msgr2=0x7f068c06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.075+0000 7f0693d5e700 1 -- 172.21.15.116:0/1933833212 shutdown_connections 2023-11-08T08:19:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.075+0000 7f0693d5e700 1 -- 172.21.15.116:0/1933833212 wait complete. 2023-11-08T08:19:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.076+0000 7f0693d5e700 1 Processor -- start 2023-11-08T08:19:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.076+0000 7f0693d5e700 1 -- start start 2023-11-08T08:19:50.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0693d5e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06780043f0 0x7f068c0b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0693d5e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f068c1508d0 0x7f068c0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0693d5e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 0x7f068c151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0693d5e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0ccc00 con 0x7f068c150fd0 2023-11-08T08:19:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0693d5e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c1496b0 con 0x7f06780043f0 2023-11-08T08:19:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0693d5e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c1519b0 con 0x7f068c1508d0 2023-11-08T08:19:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 0x7f068c151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 0x7f068c151470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60280/0 (socket says 172.21.15.116:60280) 2023-11-08T08:19:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 -- 172.21.15.116:0/2008925308 learned_addr learned my addr 172.21.15.116:0/2008925308 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0691afa700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06780043f0 0x7f068c0b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 -- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f068c1508d0 msgr2=0x7f068c0cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f068c1508d0 0x7f068c0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 -- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06780043f0 msgr2=0x7f068c0b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06780043f0 0x7f068c0b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f06922fb700 1 -- 172.21.15.116:0/2008925308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c15abc0 con 0x7f068c150fd0 2023-11-08T08:19:50.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.077+0000 7f0691afa700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06780043f0 0x7f068c0b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:50.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.078+0000 7f06922fb700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 0x7f068c151470 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f068c0d1e30 tx=0x7f06840114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.078+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400e070 con 0x7f068c150fd0 2023-11-08T08:19:50.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.078+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684003c30 con 0x7f068c150fd0 2023-11-08T08:19:50.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.078+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068c15ec60 con 0x7f068c150fd0 2023-11-08T08:19:50.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.078+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684003de0 con 0x7f068c150fd0 2023-11-08T08:19:50.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.078+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c15e490 con 0x7f068c150fd0 2023-11-08T08:19:50.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.079+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c06b380 con 0x7f068c150fd0 2023-11-08T08:19:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.079+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0684011a40 con 0x7f068c150fd0 2023-11-08T08:19:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.079+0000 7f06827fc700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f06700682f0 0x7f067006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.080+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0684086260 con 0x7f068c150fd0 2023-11-08T08:19:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.080+0000 7f0691afa700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f06700682f0 0x7f067006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.080+0000 7f0691afa700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f06700682f0 0x7f067006a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f068c14d2c0 tx=0x7f0688009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.084+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f068c06b380 con 0x7f068c150fd0 2023-11-08T08:19:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.306+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f068c06b380 con 0x7f068c150fd0 2023-11-08T08:19:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.307+0000 7f06827fc700 1 -- 172.21.15.116:0/2008925308 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f068c06b380 con 0x7f068c150fd0 2023-11-08T08:19:50.307 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:50.308 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:19:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f06700682f0 msgr2=0x7f067006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f06700682f0 0x7f067006a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f068c14d2c0 tx=0x7f0688009040 comp rx=0 tx=0).stop 2023-11-08T08:19:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 msgr2=0x7f068c151470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 0x7f068c151470 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f068c0d1e30 tx=0x7f06840114b0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 shutdown_connections 2023-11-08T08:19:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f068c150fd0 0x7f068c151470 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f06700682f0 0x7f067006a7a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06780043f0 0x7f068c0b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 --2- 172.21.15.116:0/2008925308 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f068c1508d0 0x7f068c0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 >> 172.21.15.116:0/2008925308 conn(0x7f068c06a270 msgr2=0x7f068c1543a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 shutdown_connections 2023-11-08T08:19:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.309+0000 7f0693d5e700 1 -- 172.21.15.116:0/2008925308 wait complete. 2023-11-08T08:19:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:19:50.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-08T08:19:45.064539+0000', 'modified': '2023-11-08T08:19:48.075173+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5265': {'gid': 5265, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6837/2464372713', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 2464372713}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2464372713}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-08T08:19:50.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-08T08:19:50.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-08T08:19:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.849+0000 7f863bc9d700 1 Processor -- start 2023-11-08T08:19:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.849+0000 7f863bc9d700 1 -- start start 2023-11-08T08:19:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863bc9d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8634159710 0x7f8634070370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863bc9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f86340c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863bc9d700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863406a310 con 0x7f86340cee40 2023-11-08T08:19:50.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863bc9d700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86340cf2c0 con 0x7f8634159710 2023-11-08T08:19:50.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863bc9d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86340ca410 con 0x7f86340c9a50 2023-11-08T08:19:50.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639238700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f86340cee40 0x7f86340cdff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36582/0 (socket says 172.21.15.116:36582) 2023-11-08T08:19:50.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639238700 1 -- 172.21.15.116:0/3882755425 learned_addr learned my addr 172.21.15.116:0/3882755425 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:50.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639a39700 1 --2- 172.21.15.116:0/3882755425 >> v2:172.21.15.129:3300/0 conn(0x7f8634159710 0x7f8634070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863a23a700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f86340c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8638a37700 1 -- 172.21.15.116:0/3882755425 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97636540 0 0) 0x7f863406a310 con 0x7f86340cee40 2023-11-08T08:19:50.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8638a37700 1 -- 172.21.15.116:0/3882755425 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86200035b0 con 0x7f86340cee40 2023-11-08T08:19:50.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8638a37700 1 -- 172.21.15.116:0/3882755425 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4112134978 0 0) 0x7f86200035b0 con 0x7f86340cee40 2023-11-08T08:19:50.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639a39700 1 -- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 msgr2=0x7f86340c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639a39700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f86340c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639a39700 1 -- 172.21.15.116:0/3882755425 >> v1:172.21.15.116:6789/0 conn(0x7f86340cee40 legacy=0x7f86340cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f8639a39700 1 -- 172.21.15.116:0/3882755425 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86340ce7e0 con 0x7f8634159710 2023-11-08T08:19:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.850+0000 7f863a23a700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f86340c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f8639a39700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8634159710 0x7f8634070370 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8630009870 tx=0x7f8630009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0d30b7ffbe64382 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f8638a37700 1 -- 172.21.15.116:0/3882755425 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624001670 con 0x7f8634159710 2023-11-08T08:19:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f8638a37700 1 -- 172.21.15.116:0/3882755425 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8630004030 con 0x7f8634159710 2023-11-08T08:19:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f8638a37700 1 -- 172.21.15.116:0/3882755425 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86300043a0 con 0x7f8634159710 2023-11-08T08:19:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 -- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8634159710 msgr2=0x7f8634070370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8634159710 0x7f8634070370 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8630009870 tx=0x7f8630009b80 comp rx=0 tx=0).stop 2023-11-08T08:19:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 -- 172.21.15.116:0/3882755425 shutdown_connections 2023-11-08T08:19:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8634159710 0x7f8634070370 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 --2- 172.21.15.116:0/3882755425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f86340c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 -- 172.21.15.116:0/3882755425 >> 172.21.15.116:0/3882755425 conn(0x7f86340cf400 msgr2=0x7f86340cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 -- 172.21.15.116:0/3882755425 shutdown_connections 2023-11-08T08:19:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.851+0000 7f863bc9d700 1 -- 172.21.15.116:0/3882755425 wait complete. 2023-11-08T08:19:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 Processor -- start 2023-11-08T08:19:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 -- start start 2023-11-08T08:19:50.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f863414f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86340cee40 0x7f863415a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 0x7f86341e6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863415bc60 con 0x7f8634159710 2023-11-08T08:19:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86340b2250 con 0x7f86340cee40 2023-11-08T08:19:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863bc9d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86341f60d0 con 0x7f86340c9a50 2023-11-08T08:19:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863a23a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 0x7f86341e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863a23a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 0x7f86341e6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60284/0 (socket says 172.21.15.116:60284) 2023-11-08T08:19:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.853+0000 7f863a23a700 1 -- 172.21.15.116:0/1240297091 learned_addr learned my addr 172.21.15.116:0/1240297091 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f8639a39700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f863414f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f8639238700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86340cee40 0x7f863415a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863a23a700 1 -- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 msgr2=0x7f863414f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863a23a700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f863414f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863a23a700 1 -- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86340cee40 msgr2=0x7f863415a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863a23a700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86340cee40 0x7f863415a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863a23a700 1 -- 172.21.15.116:0/1240297091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c009a70 con 0x7f8634159710 2023-11-08T08:19:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f8639a39700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f863414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f8639238700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86340cee40 0x7f863415a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863a23a700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 0x7f86341e6180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f862c00efc0 tx=0x7f862c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f862a7fc700 1 -- 172.21.15.116:0/1240297091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c009cd0 con 0x7f8634159710 2023-11-08T08:19:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f862a7fc700 1 -- 172.21.15.116:0/1240297091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c00f040 con 0x7f8634159710 2023-11-08T08:19:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f862a7fc700 1 -- 172.21.15.116:0/1240297091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c00cd60 con 0x7f8634159710 2023-11-08T08:19:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86300094a0 con 0x7f8634159710 2023-11-08T08:19:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.854+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863406a890 con 0x7f8634159710 2023-11-08T08:19:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.855+0000 7f862a7fc700 1 -- 172.21.15.116:0/1240297091 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f862c014480 con 0x7f8634159710 2023-11-08T08:19:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.856+0000 7f862a7fc700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86180682f0 0x7f861806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.856+0000 7f862a7fc700 1 -- 172.21.15.116:0/1240297091 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f862c0865d0 con 0x7f8634159710 2023-11-08T08:19:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.856+0000 7f8639a39700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86180682f0 0x7f861806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.856+0000 7f8639a39700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86180682f0 0x7f861806a7a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f863415cd00 tx=0x7f8630008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.858+0000 7f8647e3cb80 1 Processor -- start 2023-11-08T08:19:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.858+0000 7f8647e3cb80 1 -- start start 2023-11-08T08:19:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 0x55b3e9b979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bde2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9b98b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9b15de0 con 0x55b3e9bddf10 2023-11-08T08:19:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9a82140 con 0x55b3e9bd4a80 2023-11-08T08:19:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9af0db0 con 0x55b3e9bc0620 2023-11-08T08:19:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 0x55b3e9b979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 0x55b3e9b979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60298/0 (socket says 172.21.15.116:60298) 2023-11-08T08:19:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 -- 172.21.15.116:0/3796221163 learned_addr learned my addr 172.21.15.116:0/3796221163 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639238700 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bde2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f863a23a700 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9b98b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 -- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 msgr2=0x55b3e9b98b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9b98b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 -- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bd4a80 msgr2=0x55b3e9bde2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bde2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 -- 172.21.15.116:0/3796221163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3e98de790 con 0x55b3e9bddf10 2023-11-08T08:19:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.859+0000 7f8639a39700 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 0x55b3e9b979b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8630003720 tx=0x7f8630010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fac10715430c4e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f86177fe700 1 -- 172.21.15.116:0/3796221163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630003ea0 con 0x55b3e9bddf10 2023-11-08T08:19:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f86177fe700 1 -- 172.21.15.116:0/3796221163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863002a070 con 0x55b3e9bddf10 2023-11-08T08:19:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f86177fe700 1 -- 172.21.15.116:0/3796221163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630011830 con 0x55b3e9bddf10 2023-11-08T08:19:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3796221163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 msgr2=0x55b3e9b979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 0x55b3e9b979b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8630003720 tx=0x7f8630010000 comp rx=0 tx=0).stop 2023-11-08T08:19:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3796221163 shutdown_connections 2023-11-08T08:19:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9b98b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bddf10 0x55b3e9b979b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3796221163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bde2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3796221163 >> 172.21.15.116:0/3796221163 conn(0x55b3e9b8dce0 msgr2=0x55b3e9b96ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3796221163 shutdown_connections 2023-11-08T08:19:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.860+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3796221163 wait complete. 2023-11-08T08:19:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.861+0000 7f8647e3cb80 1 -- start start 2023-11-08T08:19:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.861+0000 7f8647e3cb80 1 Processor -- start 2023-11-08T08:19:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9bba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bd58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bdba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9aea120 con 0x55b3e9bd4a80 2023-11-08T08:19:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e98df390 con 0x55b3e9bc0620 2023-11-08T08:19:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9a82340 con 0x55b3e9bddf10 2023-11-08T08:19:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8647e3cb80 10 client.0 _ll_register_callbacks cb 0x55b3e9a88400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:19:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bd58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bd58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60300/0 (socket says 172.21.15.116:60300) 2023-11-08T08:19:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 -- 172.21.15.116:0/3035992651 learned_addr learned my addr 172.21.15.116:0/3035992651 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639a39700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9bba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f863a23a700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bdba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 -- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 msgr2=0x55b3e9bdba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bdba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 -- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 msgr2=0x55b3e9bba1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9bba1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 -- 172.21.15.116:0/3035992651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8630020070 con 0x55b3e9bd4a80 2023-11-08T08:19:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f863a23a700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bdba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8639238700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bd58b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f86240104c0 tx=0x7f86240107d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baca026c5da505a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.862+0000 7f8614ff9700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:19:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8614ff9700 1 -- 172.21.15.116:0/3035992651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86240135c0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8614ff9700 1 -- 172.21.15.116:0/3035992651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8624015c30 con 0x55b3e9bd4a80 2023-11-08T08:19:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3e9aee030 con 0x55b3e9bd4a80 2023-11-08T08:19:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8614ff9700 1 -- 172.21.15.116:0/3035992651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8624015df0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 10 client.5466 fetch_fsmap learned FSMap version 16 2023-11-08T08:19:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55b3e99cb0a0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8614ff9700 1 -- 172.21.15.116:0/3035992651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624008020 con 0x55b3e9bd4a80 2023-11-08T08:19:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8614ff9700 1 -- 172.21.15.116:0/3035992651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 16) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f862401d070 con 0x55b3e9bd4a80 2023-11-08T08:19:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 10 client.5466 fetch_fsmap finished waiting for FSMap version 16 2023-11-08T08:19:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 10 client.5466 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-08T08:19:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 20 client.5466 populate_metadata read hostname 'smithi116' 2023-11-08T08:19:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd88d0 0x55b3e9bd8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 4 client.5466 mds_command: new command op to 5265 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:19:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 --> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3e9939ef0 con 0x55b3e9bd88d0 2023-11-08T08:19:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8639a39700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd88d0 0x55b3e9bd8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.863+0000 7f8614ff9700 10 client.5466 ms_handle_connect on v2:172.21.15.116:6835/2464372713 2023-11-08T08:19:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8639a39700 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd88d0 0x55b3e9bd8ca0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb008ed3b74346d3 server_cookie=6650136a8d16cd7a in_seq=0 out_seq=0 2023-11-08T08:19:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8614ff9700 1 -- 172.21.15.116:0/3035992651 <== mds.0 v2:172.21.15.116:6835/2464372713 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b3e9939ef0 con 0x55b3e9bd88d0 2023-11-08T08:19:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8614ff9700 10 client.5466 handle_command_reply: tid=0 2023-11-08T08:19:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 client.5466 shutdown 2023-11-08T08:19:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 msgr2=0x55b3e9bd58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bd58b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f86240104c0 tx=0x7f86240107d0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 shutdown_connections 2023-11-08T08:19:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd88d0 0x55b3e9bd8ca0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bdba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9bba1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8614ff9700 0 client.5466 ms_handle_reset on v2:172.21.15.116:6835/2464372713 2023-11-08T08:19:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/3035992651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bd58b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 >> 172.21.15.116:0/3035992651 conn(0x55b3e9b8dce0 msgr2=0x55b3e9b93630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 shutdown_connections 2023-11-08T08:19:50.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.864+0000 7f8647e3cb80 1 -- 172.21.15.116:0/3035992651 wait complete. 2023-11-08T08:19:50.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.865+0000 7f8647e3cb80 20 client.5466 trim_cache size 0 max 16384 2023-11-08T08:19:50.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.881+0000 7f8647e3cb80 1 Processor -- start 2023-11-08T08:19:50.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.881+0000 7f8647e3cb80 1 -- start start 2023-11-08T08:19:50.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9c3c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bf68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bddf10 0x55b3e9bf6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9bdaef0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9bbf500 con 0x55b3e9bddf10 2023-11-08T08:19:50.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9bd6c20 con 0x55b3e9bc0620 2023-11-08T08:19:50.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bf68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bf68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60306/0 (socket says 172.21.15.116:60306) 2023-11-08T08:19:50.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 -- 172.21.15.116:0/4094238216 learned_addr learned my addr 172.21.15.116:0/4094238216 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:50.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 -- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 msgr2=0x55b3e9c3c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9c3c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 -- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bddf10 msgr2=0x55b3e9bf6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f863a23a700 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bddf10 0x55b3e9bf6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bddf10 0x55b3e9bf6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.882+0000 7f8639238700 1 -- 172.21.15.116:0/4094238216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3e9bc6fa0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8639238700 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bf68c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8624001e70 tx=0x7f8624013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f4d5bb8c277325 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8617fff700 1 -- 172.21.15.116:0/4094238216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624001b60 con 0x55b3e9bd4a80 2023-11-08T08:19:50.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8617fff700 1 -- 172.21.15.116:0/4094238216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624008040 con 0x55b3e9bd4a80 2023-11-08T08:19:50.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8617fff700 1 -- 172.21.15.116:0/4094238216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624013070 con 0x55b3e9bd4a80 2023-11-08T08:19:50.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 -- 172.21.15.116:0/4094238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 msgr2=0x55b3e9bf68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bf68c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8624001e70 tx=0x7f8624013000 comp rx=0 tx=0).stop 2023-11-08T08:19:50.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 -- 172.21.15.116:0/4094238216 shutdown_connections 2023-11-08T08:19:50.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bddf10 0x55b3e9bf6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bc0620 0x55b3e9c3c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/4094238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bf68c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 -- 172.21.15.116:0/4094238216 >> 172.21.15.116:0/4094238216 conn(0x55b3e9b8dce0 msgr2=0x55b3e9b95240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 -- 172.21.15.116:0/4094238216 shutdown_connections 2023-11-08T08:19:50.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.883+0000 7f8647e3cb80 1 -- 172.21.15.116:0/4094238216 wait complete. 2023-11-08T08:19:50.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.884+0000 7f8647e3cb80 1 -- start start 2023-11-08T08:19:50.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.884+0000 7f8647e3cb80 1 Processor -- start 2023-11-08T08:19:50.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9c3a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bb8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bb9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9bddbc0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9c443d0 con 0x55b3e9bc0620 2023-11-08T08:19:50.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9b94b20 con 0x55b3e9bddf10 2023-11-08T08:19:50.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8647e3cb80 10 client.0 _ll_register_callbacks cb 0x55b3e9a88400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:19:50.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bb8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bb8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60316/0 (socket says 172.21.15.116:60316) 2023-11-08T08:19:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 -- 172.21.15.116:0/2689597259 learned_addr learned my addr 172.21.15.116:0/2689597259 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f863a23a700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bb9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639a39700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9c3a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 -- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 msgr2=0x55b3e9bb9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bb9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 -- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 msgr2=0x55b3e9c3a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:50.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9c3a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 -- 172.21.15.116:0/2689597259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b3e99cb0a0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8639238700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bb8d50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8624001e70 tx=0x7f8624000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c657902fa41b582b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:50.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8614ff9700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:19:50.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.885+0000 7f8614ff9700 1 -- 172.21.15.116:0/2689597259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862402d070 con 0x55b3e9bd4a80 2023-11-08T08:19:50.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 1 -- 172.21.15.116:0/2689597259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8624013390 con 0x55b3e9bd4a80 2023-11-08T08:19:50.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f863002a070 con 0x55b3e9bd4a80 2023-11-08T08:19:50.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 1 -- 172.21.15.116:0/2689597259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8624013550 con 0x55b3e9bd4a80 2023-11-08T08:19:50.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 10 client.5472 fetch_fsmap learned FSMap version 16 2023-11-08T08:19:50.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55b3e9a97b10 con 0x55b3e9bd4a80 2023-11-08T08:19:50.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 1 -- 172.21.15.116:0/2689597259 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863002a070 con 0x55b3e9bd4a80 2023-11-08T08:19:50.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 1 -- 172.21.15.116:0/2689597259 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 16) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f86240104f0 con 0x55b3e9bd4a80 2023-11-08T08:19:50.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 10 client.5472 fetch_fsmap finished waiting for FSMap version 16 2023-11-08T08:19:50.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 10 client.5472 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-08T08:19:50.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 20 client.5472 populate_metadata read hostname 'smithi116' 2023-11-08T08:19:50.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd38d0 0x55b3e9bd3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:50.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 4 client.5472 mds_command: new command op to 5265 tid=0[{"prefix": "status"}] 2023-11-08T08:19:50.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 --> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b3e996f040 con 0x55b3e9bd38d0 2023-11-08T08:19:50.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8639a39700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd38d0 0x55b3e9bd3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:50.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 10 client.5472 ms_handle_connect on v2:172.21.15.116:6835/2464372713 2023-11-08T08:19:50.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8639a39700 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd38d0 0x55b3e9bd3ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d393d9f7d1530b4 server_cookie=5727929971764cde in_seq=0 out_seq=0 2023-11-08T08:19:50.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 1 -- 172.21.15.116:0/2689597259 <== mds.0 v2:172.21.15.116:6835/2464372713 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b3e996f040 con 0x55b3e9bd38d0 2023-11-08T08:19:50.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.886+0000 7f8614ff9700 10 client.5472 handle_command_reply: tid=0 2023-11-08T08:19:50.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 client.5472 shutdown 2023-11-08T08:19:50.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 msgr2=0x55b3e9bb8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bb8d50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8624001e70 tx=0x7f8624000f40 comp rx=0 tx=0).stop 2023-11-08T08:19:50.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 shutdown_connections 2023-11-08T08:19:50.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:6835/2464372713,v1:172.21.15.116:6837/2464372713] conn(0x55b3e9bd38d0 0x55b3e9bd3ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55b3e9bddf10 0x55b3e9bb9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8614ff9700 0 client.5472 ms_handle_reset on v2:172.21.15.116:6835/2464372713 2023-11-08T08:19:50.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3e9bc0620 0x55b3e9c3a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 --2- 172.21.15.116:0/2689597259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3e9bd4a80 0x55b3e9bb8d50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 >> 172.21.15.116:0/2689597259 conn(0x55b3e9b8dce0 msgr2=0x55b3e9bba3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 shutdown_connections 2023-11-08T08:19:50.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 1 -- 172.21.15.116:0/2689597259 wait complete. 2023-11-08T08:19:50.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f8647e3cb80 20 client.5472 trim_cache size 0 max 16384 2023-11-08T08:19:50.915 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:19:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "2f470591-d661-4b50-b7a5-062dfcf46b1d", 2023-11-08T08:19:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-08T08:19:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5265, 2023-11-08T08:19:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-08T08:19:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-08T08:19:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-08T08:19:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 5.8206408300000003, 2023-11-08T08:19:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 16, 2023-11-08T08:19:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 32, 2023-11-08T08:19:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 32, 2023-11-08T08:19:50.918 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 13.148428137 2023-11-08T08:19:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:19:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86180682f0 msgr2=0x7f861806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86180682f0 0x7f861806a7a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f863415cd00 tx=0x7f8630008030 comp rx=0 tx=0).stop 2023-11-08T08:19:50.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 msgr2=0x7f86341e6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:50.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 0x7f86341e6180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f862c00efc0 tx=0x7f862c00c3a0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 shutdown_connections 2023-11-08T08:19:50.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8634159710 0x7f86341e6180 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f86340c9a50 0x7f863414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f86180682f0 0x7f861806a7a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 --2- 172.21.15.116:0/1240297091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86340cee40 0x7f863415a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:50.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 >> 172.21.15.116:0/1240297091 conn(0x7f86340cf400 msgr2=0x7f86340cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:50.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 shutdown_connections 2023-11-08T08:19:50.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:50.887+0000 7f863bc9d700 1 -- 172.21.15.116:0/1240297091 wait complete. 2023-11-08T08:19:50.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:19:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.042+0000 7fbcb2d84700 1 Processor -- start 2023-11-08T08:19:51.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.042+0000 7fbcb2d84700 1 -- start start 2023-11-08T08:19:51.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb2d84700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcac070d90 0x7fbcac0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:51.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb2d84700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac15e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb2d84700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcac151a50 con 0x7fbcac06a270 2023-11-08T08:19:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb2d84700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcac15b440 con 0x7fbcac070d90 2023-11-08T08:19:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb2d84700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac071ac0 con 0x7fbcac15e0b0 2023-11-08T08:19:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcabfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcac070d90 0x7fbcac0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb0b20700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbcac06a270 0x7fbcac06a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36590/0 (socket says 172.21.15.116:36590) 2023-11-08T08:19:51.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcb0b20700 1 -- 172.21.15.116:0/795292320 learned_addr learned my addr 172.21.15.116:0/795292320 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:51.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcabfff700 1 -- 172.21.15.116:0/795292320 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 msgr2=0x7fbcac15e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:51.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcabfff700 1 --2- 172.21.15.116:0/795292320 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac15e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcabfff700 1 -- 172.21.15.116:0/795292320 >> v1:172.21.15.116:6789/0 conn(0x7fbcac06a270 legacy=0x7fbcac06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:51.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcabfff700 1 -- 172.21.15.116:0/795292320 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac15adf0 con 0x7fbcac070d90 2023-11-08T08:19:51.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcab7fe700 1 -- 172.21.15.116:0/795292320 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442340688 0 0) 0x7fbcac151a50 con 0x7fbcac06a270 2023-11-08T08:19:51.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.043+0000 7fbcabfff700 1 --2- 172.21.15.116:0/795292320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac070d90 0x7fbcac0711e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009840 tx=0x7fbc98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4481db3140b4bc08 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:51.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcab7fe700 1 -- 172.21.15.116:0/795292320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98011830 con 0x7fbcac070d90 2023-11-08T08:19:51.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcab7fe700 1 -- 172.21.15.116:0/795292320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc98011990 con 0x7fbcac070d90 2023-11-08T08:19:51.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcab7fe700 1 -- 172.21.15.116:0/795292320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98011d20 con 0x7fbcac070d90 2023-11-08T08:19:51.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 -- 172.21.15.116:0/795292320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac070d90 msgr2=0x7fbcac0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:51.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/795292320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac070d90 0x7fbcac0711e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009840 tx=0x7fbc98009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:51.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 -- 172.21.15.116:0/795292320 shutdown_connections 2023-11-08T08:19:51.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/795292320 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac15e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/795292320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac070d90 0x7fbcac0711e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 -- 172.21.15.116:0/795292320 >> 172.21.15.116:0/795292320 conn(0x7fbcac06c010 msgr2=0x7fbcac0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:51.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 -- 172.21.15.116:0/795292320 shutdown_connections 2023-11-08T08:19:51.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.044+0000 7fbcb2d84700 1 -- 172.21.15.116:0/795292320 wait complete. 2023-11-08T08:19:51.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.045+0000 7fbcb2d84700 1 Processor -- start 2023-11-08T08:19:51.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.045+0000 7fbcb2d84700 1 -- start start 2023-11-08T08:19:51.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb2d84700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 0x7fbcac0c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:51.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb2d84700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcac070d90 0x7fbcac0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:51.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb2d84700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac1f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:51.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb2d84700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac0c4de0 con 0x7fbcac06a270 2023-11-08T08:19:51.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb2d84700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac0c50c0 con 0x7fbcac070d90 2023-11-08T08:19:51.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb2d84700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac0c53a0 con 0x7fbcac15e0b0 2023-11-08T08:19:51.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb0b20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 0x7fbcac0c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:51.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb0b20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 0x7fbcac0c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60320/0 (socket says 172.21.15.116:60320) 2023-11-08T08:19:51.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.046+0000 7fbcb0b20700 1 -- 172.21.15.116:0/341887000 learned_addr learned my addr 172.21.15.116:0/341887000 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:51.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb1321700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac1f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:51.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcabfff700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcac070d90 0x7fbcac0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb0b20700 1 -- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 msgr2=0x7fbcac1f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb0b20700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac1f3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb0b20700 1 -- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcac070d90 msgr2=0x7fbcac0c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb0b20700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcac070d90 0x7fbcac0c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb0b20700 1 -- 172.21.15.116:0/341887000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980094a0 con 0x7fbcac06a270 2023-11-08T08:19:51.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb0b20700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 0x7fbcac0c5e80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbca00011e0 tx=0x7fbca000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:51.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0004c80 con 0x7fbcac06a270 2023-11-08T08:19:51.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca0004e00 con 0x7fbcac06a270 2023-11-08T08:19:51.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac15aac0 con 0x7fbcac06a270 2023-11-08T08:19:51.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0007310 con 0x7fbcac06a270 2023-11-08T08:19:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.047+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac15b0d0 con 0x7fbcac06a270 2023-11-08T08:19:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.048+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac071720 con 0x7fbcac06a270 2023-11-08T08:19:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.049+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbca001a020 con 0x7fbcac06a270 2023-11-08T08:19:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.049+0000 7fbca97fa700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc900682f0 0x7fbc9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.049+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbca0085ef0 con 0x7fbcac06a270 2023-11-08T08:19:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.049+0000 7fbcabfff700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc900682f0 0x7fbc9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.049+0000 7fbcabfff700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc900682f0 0x7fbc9006a7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbc980090b0 tx=0x7fbc98008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.053+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbca00528a0 con 0x7fbcac06a270 2023-11-08T08:19:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.275+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbcac0743f0 con 0x7fbcac06a270 2023-11-08T08:19:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.275+0000 7fbca97fa700 1 -- 172.21.15.116:0/341887000 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbca0055ec0 con 0x7fbcac06a270 2023-11-08T08:19:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc900682f0 msgr2=0x7fbc9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc900682f0 0x7fbc9006a7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbc980090b0 tx=0x7fbc98008030 comp rx=0 tx=0).stop 2023-11-08T08:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 msgr2=0x7fbcac0c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 0x7fbcac0c5e80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbca00011e0 tx=0x7fbca000cae0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 shutdown_connections 2023-11-08T08:19:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbcac15e0b0 0x7fbcac1f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcac06a270 0x7fbcac0c5e80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbc900682f0 0x7fbc9006a7a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 --2- 172.21.15.116:0/341887000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcac070d90 0x7fbcac0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:51.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 >> 172.21.15.116:0/341887000 conn(0x7fbcac06c010 msgr2=0x7fbcac0af850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:51.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 shutdown_connections 2023-11-08T08:19:51.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:51.277+0000 7fbcb2d84700 1 -- 172.21.15.116:0/341887000 wait complete. 2023-11-08T08:19:51.289 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:19:51.290 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:19:51.290 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-08T08:19:51.290 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-08T08:19:51.291 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:51.291 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:19:51.291 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-08T08:19:51.292 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:19:51.292 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:19:51.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.293 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-08T08:19:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:19:51.312 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:19:51.312 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:19:51.312 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.312 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:19:51.313 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:51.313 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:51.313 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-08T08:19:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:51.314 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:51.314 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-08T08:19:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:19:51.319 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:19:51.319 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:51.319 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:19:51.320 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42226sec preferred_lft 42226sec 2023-11-08T08:19:51.320 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-08T08:19:51.320 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:19:51.320 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:19:51.321 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-08T08:19:51.321 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.321 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::8c0:d4ff:fe3b:7f6b/64 scope link 2023-11-08T08:19:51.321 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:19:51.322 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-08T08:19:51.322 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bc76:a3ff:fe22:41c7/64 scope link 2023-11-08T08:19:51.322 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:19:51.322 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:ab:30:29:e8:99 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-08T08:19:51.323 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bcab:30ff:fe29:e899/64 scope link 2023-11-08T08:19:51.323 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:19:51.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.324 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:19:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-08T08:19:51.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.379 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-08T08:19:51.439 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-08T08:19:51.439 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:19:51.439 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:19:51.439 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:19:51.440 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:19:51.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.440 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:19:51.441 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:19:51.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:19:51.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:19:51.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:19:51.442 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:19:51.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.543 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:19:51.562 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-08T08:19:51.562 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-08T08:19:51.562 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:19:51.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-08T08:19:51.629 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-08T08:19:51.629 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:19:51.630 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:51.686 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:19:51.687 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:19:51.687 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:51.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:19:51.812 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:19:51.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:19:51.878 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:19:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:19:51.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:51.880 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:19:51.938 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-08T08:19:51.938 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-11-08T08:19:51.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:19:52.006 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:19:52.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:19:52.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-08T08:19:52.065+0000 7f0fa7dcf5c0 -1 init, newargv = 0x55726e4d5950 newargc=15 2023-11-08T08:19:52.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[138147]: starting ceph client 2023-11-08T08:19:52.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[138147]: starting fuse 2023-11-08T08:19:52.076 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:19:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:19:52.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.078 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:19:52.136 INFO:teuthology.orchestra.run.smithi116.stdout:46 2023-11-08T08:19:52.137 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46] 2023-11-08T08:19:52.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:52.138 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-08T08:19:52.138 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-08T08:19:52.138 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:52.139 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-08T08:19:52.139 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:52.140 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-08T08:19:52.140 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-08T08:19:52.140 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-08T08:19:52.141 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-08T08:19:52.141 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:52.141 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-08T08:19:52.142 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-08T08:19:52.142 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-08T08:19:52.142 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-08T08:19:52.143 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:52.143 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-08T08:19:52.144 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:19:52.144 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:19:52.144 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:19:52.145 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-08T08:19:52.145 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-08T08:19:52.146 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-08T08:19:52.146 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:19:52.146 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:52.147 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-08T08:19:52.147 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:52.260 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.138147.asok 2023-11-08T08:19:52.442 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.138147.asok 2023-11-08T08:19:52.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.138147.asok status 2023-11-08T08:19:52.598 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:19:52.598 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-08T08:19:52.598 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:19:52.598 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:19:52.599 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-08T08:19:52.599 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-08T08:19:52.599 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-08T08:19:52.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "138147", 2023-11-08T08:19:52.599 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-08T08:19:52.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:52.600 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-08T08:19:52.600 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-08T08:19:52.600 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5487, 2023-11-08T08:19:52.600 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-08T08:19:52.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-08T08:19:52.601 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-08T08:19:52.601 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5487 2023-11-08T08:19:52.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:52.601 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:19:52.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:19:52.602 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:19:52.602 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 612839150 2023-11-08T08:19:52.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:19:52.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:52.603 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:19:52.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:19:52.603 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:19:52.603 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 612839150 2023-11-08T08:19:52.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:19:52.604 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.5487 192.168.0.1:0/612839150", 2023-11-08T08:19:52.604 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/612839150", 2023-11-08T08:19:52.604 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-08T08:19:52.604 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 16, 2023-11-08T08:19:52.605 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 32, 2023-11-08T08:19:52.605 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:19:52.605 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-08T08:19:52.605 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-08T08:19:52.605 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:19:52.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.608 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-08T08:19:52.608 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:52.627 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-08T08:19:52.627 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:52.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:52.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.697 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2023-11-08T08:19:52.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.759 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2023-11-08T08:19:52.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.820 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2023-11-08T08:19:52.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.881 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2023-11-08T08:19:52.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:52.945 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2023-11-08T08:19:53.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:53.006 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2023-11-08T08:19:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:./ 2023-11-08T08:19:53.074 INFO:teuthology.orchestra.run.smithi116.stdout:./rfile 2023-11-08T08:19:53.074 INFO:teuthology.orchestra.run.smithi116.stdout:./parentdir 2023-11-08T08:19:53.074 INFO:teuthology.orchestra.run.smithi116.stdout:./parentdir/pfile 2023-11-08T08:19:53.074 INFO:teuthology.orchestra.run.smithi116.stdout:./parentdir/childdir 2023-11-08T08:19:53.075 INFO:teuthology.orchestra.run.smithi116.stdout:./parentdir/childdir/cfile 2023-11-08T08:19:53.075 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:53.076 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:53.076 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:19:53.076 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:53.076 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2023-11-08T08:19:53.077 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:53.181 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-08T08:19:53.369 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:53.369 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:53.370 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:19:53.370 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:53.371 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2023-11-08T08:19:53.371 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:53.477 INFO:teuthology.orchestra.run.smithi116.stdout:1099511627778 2023-11-08T08:19:53.665 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:53.666 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:53.666 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:19:53.667 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:53.667 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2023-11-08T08:19:53.667 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:53.773 INFO:teuthology.orchestra.run.smithi116.stdout:1099511627776 2023-11-08T08:19:53.963 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:53.963 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:53.964 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:19:53.964 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:53.964 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2023-11-08T08:19:53.965 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:54.071 INFO:teuthology.orchestra.run.smithi116.stdout:1099511628279 2023-11-08T08:19:54.260 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:54.261 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:54.261 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:19:54.261 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:54.262 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2023-11-08T08:19:54.262 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:54.370 INFO:teuthology.orchestra.run.smithi116.stdout:1099511627777 2023-11-08T08:19:54.559 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:19:54.559 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:19:54.560 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:19:54.560 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:19:54.561 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2023-11-08T08:19:54.561 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:19:54.667 INFO:teuthology.orchestra.run.smithi116.stdout:1099511628280 2023-11-08T08:19:54.855 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:54.856 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:54.914 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:55.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.072+0000 7fedd9776700 1 Processor -- start 2023-11-08T08:19:55.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.072+0000 7fedd9776700 1 -- start start 2023-11-08T08:19:55.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd9776700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd4142db0 0x7fedd413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd9776700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fedd40b1770 0x7fedd40b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd9776700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd415e010 con 0x7fedd40ce040 2023-11-08T08:19:55.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd9776700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd40b2130 con 0x7fedd40b1770 2023-11-08T08:19:55.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd9776700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd413fc90 con 0x7fedd4142db0 2023-11-08T08:19:55.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd37fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fedd40b1770 0x7fedd40b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd27fc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd4142db0 0x7fedd413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd2ffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fedd40ce040 0x7fedd415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:38488/0 (socket says 172.21.15.116:38488) 2023-11-08T08:19:55.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.073+0000 7fedd2ffd700 1 -- 172.21.15.116:0/1887162122 learned_addr learned my addr 172.21.15.116:0/1887162122 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:55.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd37fe700 1 -- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd4142db0 msgr2=0x7fedd413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:55.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd37fe700 1 --2- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd4142db0 0x7fedd413ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd37fe700 1 -- 172.21.15.116:0/1887162122 >> v1:172.21.15.129:6789/0 conn(0x7fedd40ce040 legacy=0x7fedd415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd37fe700 1 -- 172.21.15.116:0/1887162122 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd413f6d0 con 0x7fedd40b1770 2023-11-08T08:19:55.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd37fe700 1 --2- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd40b1bf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009d90 tx=0x7fedcc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c32c9ab03e22130 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd1ffb700 1 -- 172.21.15.116:0/1887162122 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc00e070 con 0x7fedd40b1770 2023-11-08T08:19:55.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd1ffb700 1 -- 172.21.15.116:0/1887162122 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc00bb20 con 0x7fedd40b1770 2023-11-08T08:19:55.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.074+0000 7fedd1ffb700 1 -- 172.21.15.116:0/1887162122 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc012430 con 0x7fedd40b1770 2023-11-08T08:19:55.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 -- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 msgr2=0x7fedd40b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 --2- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd40b1bf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009d90 tx=0x7fedcc00b330 comp rx=0 tx=0).stop 2023-11-08T08:19:55.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 -- 172.21.15.116:0/1887162122 shutdown_connections 2023-11-08T08:19:55.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 --2- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd40b1bf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 --2- 172.21.15.116:0/1887162122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd4142db0 0x7fedd413ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 -- 172.21.15.116:0/1887162122 >> 172.21.15.116:0/1887162122 conn(0x7fedd406c010 msgr2=0x7fedd40cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 -- 172.21.15.116:0/1887162122 shutdown_connections 2023-11-08T08:19:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.075+0000 7fedd9776700 1 -- 172.21.15.116:0/1887162122 wait complete. 2023-11-08T08:19:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.076+0000 7fedd9776700 1 Processor -- start 2023-11-08T08:19:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.076+0000 7fedd9776700 1 -- start start 2023-11-08T08:19:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd9776700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd41418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd9776700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd40ce040 0x7fedd40cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd9776700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 0x7fedd40cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd9776700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40b2f00 con 0x7fedd4142db0 2023-11-08T08:19:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd9776700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40b31e0 con 0x7fedd40b1770 2023-11-08T08:19:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd9776700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40b34c0 con 0x7fedd40ce040 2023-11-08T08:19:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 0x7fedd40cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 0x7fedd40cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34968/0 (socket says 172.21.15.116:34968) 2023-11-08T08:19:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 -- 172.21.15.116:0/1395001290 learned_addr learned my addr 172.21.15.116:0/1395001290 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd2ffd700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd41418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd27fc700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd40ce040 0x7fedd40cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 -- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd40ce040 msgr2=0x7fedd40cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd40ce040 0x7fedd40cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 -- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 msgr2=0x7fedd41418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd41418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 -- 172.21.15.116:0/1395001290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc009a70 con 0x7fedd4142db0 2023-11-08T08:19:55.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.077+0000 7fedd37fe700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 0x7fedd40cb710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fedcc014040 tx=0x7fedcc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.078+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc012430 con 0x7fedd4142db0 2023-11-08T08:19:55.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.078+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc020070 con 0x7fedd4142db0 2023-11-08T08:19:55.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.078+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd415d100 con 0x7fedd4142db0 2023-11-08T08:19:55.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.078+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc003ea0 con 0x7fedd4142db0 2023-11-08T08:19:55.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.078+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd413f9b0 con 0x7fedd4142db0 2023-11-08T08:19:55.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.079+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd415d3e0 con 0x7fedd4142db0 2023-11-08T08:19:55.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.079+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fedcc02e070 con 0x7fedd4142db0 2023-11-08T08:19:55.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.079+0000 7fedbb7fe700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fedc00682f0 0x7fedc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.080+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fedcc022070 con 0x7fedd4142db0 2023-11-08T08:19:55.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.080+0000 7fedd2ffd700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fedc00682f0 0x7fedc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.080+0000 7fedd2ffd700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fedc00682f0 0x7fedc006a7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005910 tx=0x7fedc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.084+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fedcc054280 con 0x7fedd4142db0 2023-11-08T08:19:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.265+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd40743f0 con 0x7fedd4142db0 2023-11-08T08:19:55.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.266+0000 7fedbb7fe700 1 -- 172.21.15.116:0/1395001290 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+17969 (secure 0 0 0) 0x7fedcc02c090 con 0x7fedd4142db0 2023-11-08T08:19:55.267 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:55.268 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:46.066442+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:55.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fedc00682f0 msgr2=0x7fedc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fedc00682f0 0x7fedc006a7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005910 tx=0x7fedc8009040 comp rx=0 tx=0).stop 2023-11-08T08:19:55.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 msgr2=0x7fedd40cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 0x7fedd40cb710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fedcc014040 tx=0x7fedcc004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 shutdown_connections 2023-11-08T08:19:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd4142db0 0x7fedd40cb710 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fedc00682f0 0x7fedc006a7a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedd40b1770 0x7fedd41418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 --2- 172.21.15.116:0/1395001290 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fedd40ce040 0x7fedd40cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 >> 172.21.15.116:0/1395001290 conn(0x7fedd406c010 msgr2=0x7fedd4070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 shutdown_connections 2023-11-08T08:19:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.268+0000 7fedd9776700 1 -- 172.21.15.116:0/1395001290 wait complete. 2023-11-08T08:19:55.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:19:43.321551+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:19:44.023597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:55.282 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:55.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.403+0000 7f0d90091700 1 Processor -- start 2023-11-08T08:19:55.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.404+0000 7f0d90091700 1 -- start start 2023-11-08T08:19:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.404+0000 7f0d90091700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0d880cec00 0x7f0d88065fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.404+0000 7f0d90091700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0d880cac90 0x7f0d880cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.404+0000 7f0d90091700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8815ceb0 con 0x7f0d88160300 2023-11-08T08:19:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.404+0000 7f0d90091700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d880cb650 con 0x7f0d880cac90 2023-11-08T08:19:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d90091700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d881608c0 con 0x7f0d880cec00 2023-11-08T08:19:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0d880cac90 0x7f0d880cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0d880cac90 0x7f0d880cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34980/0 (socket says 172.21.15.116:34980) 2023-11-08T08:19:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 -- 172.21.15.116:0/2133180354 learned_addr learned my addr 172.21.15.116:0/2133180354 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:55.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8de2d700 1 --2- 172.21.15.116:0/2133180354 >> v2:172.21.15.129:3300/0 conn(0x7f0d880cec00 0x7f0d88065fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 -- 172.21.15.116:0/2133180354 >> v2:172.21.15.129:3300/0 conn(0x7f0d880cec00 msgr2=0x7f0d88065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 --2- 172.21.15.116:0/2133180354 >> v2:172.21.15.129:3300/0 conn(0x7f0d880cec00 0x7f0d88065fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 -- 172.21.15.116:0/2133180354 >> v1:172.21.15.129:6789/0 conn(0x7f0d88160300 legacy=0x7f0d8815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 -- 172.21.15.116:0/2133180354 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88160ba0 con 0x7f0d880cac90 2023-11-08T08:19:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8de2d700 1 --2- 172.21.15.116:0/2133180354 >> v2:172.21.15.129:3300/0 conn(0x7f0d880cec00 0x7f0d88065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8e62e700 1 --2- 172.21.15.116:0/2133180354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d880cb110 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0d880cceb0 tx=0x7f0d8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece4eb3a9e8eb38a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8ce2b700 1 -- 172.21.15.116:0/2133180354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8000e070 con 0x7f0d880cac90 2023-11-08T08:19:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8ce2b700 1 -- 172.21.15.116:0/2133180354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80003a20 con 0x7f0d880cac90 2023-11-08T08:19:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.405+0000 7f0d8ce2b700 1 -- 172.21.15.116:0/2133180354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80003dd0 con 0x7f0d880cac90 2023-11-08T08:19:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 -- 172.21.15.116:0/2133180354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 msgr2=0x7f0d880cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 --2- 172.21.15.116:0/2133180354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d880cb110 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0d880cceb0 tx=0x7f0d8000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 -- 172.21.15.116:0/2133180354 shutdown_connections 2023-11-08T08:19:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 --2- 172.21.15.116:0/2133180354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d880cb110 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 --2- 172.21.15.116:0/2133180354 >> v2:172.21.15.129:3300/0 conn(0x7f0d880cec00 0x7f0d88065fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 -- 172.21.15.116:0/2133180354 >> 172.21.15.116:0/2133180354 conn(0x7f0d880cf400 msgr2=0x7f0d880cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 -- 172.21.15.116:0/2133180354 shutdown_connections 2023-11-08T08:19:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.406+0000 7f0d90091700 1 -- 172.21.15.116:0/2133180354 wait complete. 2023-11-08T08:19:55.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.407+0000 7f0d90091700 1 Processor -- start 2023-11-08T08:19:55.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.407+0000 7f0d90091700 1 -- start start 2023-11-08T08:19:55.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d8815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0d880cec00 0x7f0d88151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d88160300 0x7f0d880712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8815ef40 con 0x7f0d880cac90 2023-11-08T08:19:55.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8815d750 con 0x7f0d88160300 2023-11-08T08:19:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8815e9c0 con 0x7f0d880cec00 2023-11-08T08:19:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d8815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d8815dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34982/0 (socket says 172.21.15.116:34982) 2023-11-08T08:19:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 -- 172.21.15.116:0/2096206940 learned_addr learned my addr 172.21.15.116:0/2096206940 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8e62e700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d88160300 0x7f0d880712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8d62c700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0d880cec00 0x7f0d88151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 -- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0d880cec00 msgr2=0x7f0d88151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0d880cec00 0x7f0d88151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 -- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d88160300 msgr2=0x7f0d880712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d88160300 0x7f0d880712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 -- 172.21.15.116:0/2096206940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80009b30 con 0x7f0d880cac90 2023-11-08T08:19:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d8de2d700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d8815dbe0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800c750 tx=0x7f0d7800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78011800 con 0x7f0d880cac90 2023-11-08T08:19:55.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d78011960 con 0x7f0d880cac90 2023-11-08T08:19:55.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78011cd0 con 0x7f0d880cac90 2023-11-08T08:19:55.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d88161160 con 0x7f0d880cac90 2023-11-08T08:19:55.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.408+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d88160e80 con 0x7f0d880cac90 2023-11-08T08:19:55.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.409+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d88066510 con 0x7f0d880cac90 2023-11-08T08:19:55.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.410+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d78020070 con 0x7f0d880cac90 2023-11-08T08:19:55.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.410+0000 7f0d7e7fc700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0d6c0682f0 0x7f0d6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.410+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0d78085930 con 0x7f0d880cac90 2023-11-08T08:19:55.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.410+0000 7f0d8d62c700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0d6c0682f0 0x7f0d6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.411+0000 7f0d8d62c700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0d6c0682f0 0x7f0d6c06a7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0d8815e7e0 tx=0x7f0d840090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.414+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d78052190 con 0x7f0d880cac90 2023-11-08T08:19:55.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.599+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0d8815da30 con 0x7f0d880cac90 2023-11-08T08:19:55.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.599+0000 7f0d7e7fc700 1 -- 172.21.15.116:0/2096206940 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0d780557b0 con 0x7f0d880cac90 2023-11-08T08:19:55.601 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0d6c0682f0 msgr2=0x7f0d6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0d6c0682f0 0x7f0d6c06a7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0d8815e7e0 tx=0x7f0d840090e0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 msgr2=0x7f0d8815dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d8815dbe0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800c750 tx=0x7f0d7800ca60 comp rx=0 tx=0).stop 2023-11-08T08:19:55.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 shutdown_connections 2023-11-08T08:19:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0d6c0682f0 0x7f0d6c06a7a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d88160300 0x7f0d880712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d880cac90 0x7f0d8815dbe0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 --2- 172.21.15.116:0/2096206940 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0d880cec00 0x7f0d88151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 >> 172.21.15.116:0/2096206940 conn(0x7f0d880cf400 msgr2=0x7f0d8806c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 shutdown_connections 2023-11-08T08:19:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.601+0000 7f0d90091700 1 -- 172.21.15.116:0/2096206940 wait complete. 2023-11-08T08:19:55.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.736+0000 7fb7fe3f4700 1 Processor -- start 2023-11-08T08:19:55.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.736+0000 7fb7fe3f4700 1 -- start start 2023-11-08T08:19:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7fe3f4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7f80b1650 0x7fb7f80b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7fe3f4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f81599b0 0x7fb7f815d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7fe3f4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f8159ec0 con 0x7fb7f80af0a0 2023-11-08T08:19:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7fe3f4700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f815aa60 con 0x7fb7f80b1650 2023-11-08T08:19:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7fe3f4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80b1f60 con 0x7fb7f81599b0 2023-11-08T08:19:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7f80b1650 0x7fb7f80b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7f80b1650 0x7fb7f80b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34998/0 (socket says 172.21.15.116:34998) 2023-11-08T08:19:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 -- 172.21.15.116:0/717929887 learned_addr learned my addr 172.21.15.116:0/717929887 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f77fe700 1 --2- 172.21.15.116:0/717929887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f81599b0 0x7fb7f815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 -- 172.21.15.116:0/717929887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f81599b0 msgr2=0x7fb7f815d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 --2- 172.21.15.116:0/717929887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f81599b0 0x7fb7f815d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 -- 172.21.15.116:0/717929887 >> v1:172.21.15.129:6789/0 conn(0x7fb7f80af0a0 legacy=0x7fb7f815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 -- 172.21.15.116:0/717929887 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f815d8a0 con 0x7fb7f80b1650 2023-11-08T08:19:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.737+0000 7fb7f7fff700 1 --2- 172.21.15.116:0/717929887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80b1650 0x7fb7f80b1a20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0098f0 tx=0x7fb7ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a6071ce5a55c4e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7f6ffd700 1 -- 172.21.15.116:0/717929887 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec004230 con 0x7fb7f80b1650 2023-11-08T08:19:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7f6ffd700 1 -- 172.21.15.116:0/717929887 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec004390 con 0x7fb7f80b1650 2023-11-08T08:19:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7f6ffd700 1 -- 172.21.15.116:0/717929887 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0036a0 con 0x7fb7f80b1650 2023-11-08T08:19:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/717929887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80b1650 msgr2=0x7fb7f80b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/717929887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80b1650 0x7fb7f80b1a20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0098f0 tx=0x7fb7ec009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/717929887 shutdown_connections 2023-11-08T08:19:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/717929887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80b1650 0x7fb7f80b1a20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/717929887 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f81599b0 0x7fb7f815d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/717929887 >> 172.21.15.116:0/717929887 conn(0x7fb7f806c010 msgr2=0x7fb7f80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/717929887 shutdown_connections 2023-11-08T08:19:55.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.738+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/717929887 wait complete. 2023-11-08T08:19:55.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.739+0000 7fb7fe3f4700 1 Processor -- start 2023-11-08T08:19:55.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 -- start start 2023-11-08T08:19:55.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 0x7fb7f814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f80b1650 0x7fb7f80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8158220 0x7fb7f81f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c7660 con 0x7fb7f80af0a0 2023-11-08T08:19:55.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80cc0b0 con 0x7fb7f8158220 2023-11-08T08:19:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fe3f4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c7850 con 0x7fb7f80b1650 2023-11-08T08:19:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7f7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 0x7fb7f814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7f7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 0x7fb7f814f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35004/0 (socket says 172.21.15.116:35004) 2023-11-08T08:19:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7f7fff700 1 -- 172.21.15.116:0/3821728257 learned_addr learned my addr 172.21.15.116:0/3821728257 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7fc991700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8158220 0x7fb7f81f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.740+0000 7fb7f77fe700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f80b1650 0x7fb7f80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f7fff700 1 -- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f80b1650 msgr2=0x7fb7f80cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f7fff700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f80b1650 0x7fb7f80cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f7fff700 1 -- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8158220 msgr2=0x7fb7f81f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f7fff700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8158220 0x7fb7f81f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f7fff700 1 -- 172.21.15.116:0/3821728257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec009550 con 0x7fb7f80af0a0 2023-11-08T08:19:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f77fe700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f80b1650 0x7fb7f80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7fc991700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8158220 0x7fb7f81f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:19:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f7fff700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 0x7fb7f814f1f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009130 tx=0x7fb7ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec00b860 con 0x7fb7f80af0a0 2023-11-08T08:19:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec00b9c0 con 0x7fb7f80af0a0 2023-11-08T08:19:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f815de60 con 0x7fb7f80af0a0 2023-11-08T08:19:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec00bd70 con 0x7fb7f80af0a0 2023-11-08T08:19:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.741+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f815db80 con 0x7fb7f80af0a0 2023-11-08T08:19:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.742+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f815d590 con 0x7fb7f80af0a0 2023-11-08T08:19:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.742+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7ec017020 con 0x7fb7f80af0a0 2023-11-08T08:19:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.743+0000 7fb7f4ff9700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7e40682f0 0x7fb7e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.743+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb7ec0875a0 con 0x7fb7f80af0a0 2023-11-08T08:19:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.743+0000 7fb7f77fe700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7e40682f0 0x7fb7e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.743+0000 7fb7f77fe700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7e40682f0 0x7fb7e406a7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80b5740 tx=0x7fb7e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.747+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7ec053e00 con 0x7fb7f80af0a0 2023-11-08T08:19:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.927+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7f80743f0 con 0x7fb7f80af0a0 2023-11-08T08:19:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.928+0000 7fb7f4ff9700 1 -- 172.21.15.116:0/3821728257 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7ec057420 con 0x7fb7f80af0a0 2023-11-08T08:19:55.930 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:55.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7e40682f0 msgr2=0x7fb7e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7e40682f0 0x7fb7e406a7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80b5740 tx=0x7fb7e8008040 comp rx=0 tx=0).stop 2023-11-08T08:19:55.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 msgr2=0x7fb7f814f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:55.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 0x7fb7f814f1f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009130 tx=0x7fb7ec004750 comp rx=0 tx=0).stop 2023-11-08T08:19:55.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 shutdown_connections 2023-11-08T08:19:55.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7e40682f0 0x7fb7e406a7a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8158220 0x7fb7f81f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f80af0a0 0x7fb7f814f1f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.929+0000 7fb7fe3f4700 1 --2- 172.21.15.116:0/3821728257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f80b1650 0x7fb7f80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:55.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.930+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 >> 172.21.15.116:0/3821728257 conn(0x7fb7f806c010 msgr2=0x7fb7f806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:55.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.930+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 shutdown_connections 2023-11-08T08:19:55.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:55.930+0000 7fb7fe3f4700 1 -- 172.21.15.116:0/3821728257 wait complete. 2023-11-08T08:19:55.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.065+0000 7f47e580e700 1 Processor -- start 2023-11-08T08:19:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.065+0000 7f47e580e700 1 -- start start 2023-11-08T08:19:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47e580e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 0x7f47e00d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47e580e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 0x7f47e00ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47e580e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e015a9c0 con 0x7f47e015a5f0 2023-11-08T08:19:56.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47e580e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e00c7b80 con 0x7f47e00cc200 2023-11-08T08:19:56.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47e580e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00cc990 con 0x7f47e00b2ee0 2023-11-08T08:19:56.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47de7fc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 0x7f47e00d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47df7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 0x7f47e00ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47df7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 0x7f47e00ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:54962/0 (socket says 172.21.15.116:54962) 2023-11-08T08:19:56.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47deffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f47e015a5f0 0x7f47e00b27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:38514/0 (socket says 172.21.15.116:38514) 2023-11-08T08:19:56.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47de7fc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 0x7f47e00d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:47250/0 (socket says 172.21.15.116:47250) 2023-11-08T08:19:56.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.066+0000 7f47deffd700 1 -- 172.21.15.116:0/679571657 learned_addr learned my addr 172.21.15.116:0/679571657 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:56.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47de7fc700 1 -- 172.21.15.116:0/679571657 >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 msgr2=0x7f47e00ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:56.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47de7fc700 1 --2- 172.21.15.116:0/679571657 >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 0x7f47e00ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47de7fc700 1 -- 172.21.15.116:0/679571657 >> v1:172.21.15.129:6789/0 conn(0x7f47e015a5f0 legacy=0x7f47e00b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47de7fc700 1 -- 172.21.15.116:0/679571657 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00ccc70 con 0x7f47e00b2ee0 2023-11-08T08:19:56.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47df7fe700 1 --2- 172.21.15.116:0/679571657 >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 0x7f47e00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:56.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47de7fc700 1 --2- 172.21.15.116:0/679571657 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 0x7f47e00d0ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009840 tx=0x7f47d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3107c4ad25e41699 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47ddffb700 1 -- 172.21.15.116:0/679571657 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0011830 con 0x7f47e00b2ee0 2023-11-08T08:19:56.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47ddffb700 1 -- 172.21.15.116:0/679571657 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d0011990 con 0x7f47e00b2ee0 2023-11-08T08:19:56.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.067+0000 7f47ddffb700 1 -- 172.21.15.116:0/679571657 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0011d20 con 0x7f47e00b2ee0 2023-11-08T08:19:56.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 -- 172.21.15.116:0/679571657 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 msgr2=0x7f47e00d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 --2- 172.21.15.116:0/679571657 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 0x7f47e00d0ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009840 tx=0x7f47d0009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:56.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 -- 172.21.15.116:0/679571657 shutdown_connections 2023-11-08T08:19:56.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 --2- 172.21.15.116:0/679571657 >> v2:172.21.15.129:3300/0 conn(0x7f47e00cc200 0x7f47e00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 --2- 172.21.15.116:0/679571657 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00b2ee0 0x7f47e00d0ad0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 -- 172.21.15.116:0/679571657 >> 172.21.15.116:0/679571657 conn(0x7f47e006ea80 msgr2=0x7f47e006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:56.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 -- 172.21.15.116:0/679571657 shutdown_connections 2023-11-08T08:19:56.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.068+0000 7f47e580e700 1 -- 172.21.15.116:0/679571657 wait complete. 2023-11-08T08:19:56.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.069+0000 7f47e580e700 1 Processor -- start 2023-11-08T08:19:56.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.069+0000 7f47e580e700 1 -- start start 2023-11-08T08:19:56.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47e580e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e00b2ee0 0x7f47e0151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47e580e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00cc200 0x7f47e00cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47e580e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 0x7f47e00cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47e580e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e01518b0 con 0x7f47e015a5f0 2023-11-08T08:19:56.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47e580e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00c6810 con 0x7f47e00b2ee0 2023-11-08T08:19:56.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47e580e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00c6af0 con 0x7f47e00cc200 2023-11-08T08:19:56.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47df7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 0x7f47e00cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47df7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 0x7f47e00cf170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35012/0 (socket says 172.21.15.116:35012) 2023-11-08T08:19:56.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47df7fe700 1 -- 172.21.15.116:0/89442079 learned_addr learned my addr 172.21.15.116:0/89442079 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:56.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47de7fc700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00cc200 0x7f47e00cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.070+0000 7f47deffd700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e00b2ee0 0x7f47e0151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47df7fe700 1 -- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00cc200 msgr2=0x7f47e00cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47df7fe700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00cc200 0x7f47e00cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47df7fe700 1 -- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e00b2ee0 msgr2=0x7f47e0151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47df7fe700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e00b2ee0 0x7f47e0151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47df7fe700 1 -- 172.21.15.116:0/89442079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d8009a70 con 0x7f47e015a5f0 2023-11-08T08:19:56.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47df7fe700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 0x7f47e00cf170 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f47d800efc0 tx=0x7f47d800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d800ca90 con 0x7f47e015a5f0 2023-11-08T08:19:56.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d800f040 con 0x7f47e015a5f0 2023-11-08T08:19:56.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d00094a0 con 0x7f47e015a5f0 2023-11-08T08:19:56.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d800ce10 con 0x7f47e015a5f0 2023-11-08T08:19:56.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.071+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e00cd230 con 0x7f47e015a5f0 2023-11-08T08:19:56.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.072+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e00ccf50 con 0x7f47e015a5f0 2023-11-08T08:19:56.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.073+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47d8003a20 con 0x7f47e015a5f0 2023-11-08T08:19:56.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.073+0000 7f47c7fff700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47cc0682f0 0x7f47cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.073+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f47d8086f50 con 0x7f47e015a5f0 2023-11-08T08:19:56.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.073+0000 7f47deffd700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47cc0682f0 0x7f47cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.073+0000 7f47deffd700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47cc0682f0 0x7f47cc06a7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47d4005910 tx=0x7f47d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.077+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47d80537c0 con 0x7f47e015a5f0 2023-11-08T08:19:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.254+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f47e0151ae0 con 0x7f47e015a5f0 2023-11-08T08:19:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.255+0000 7f47c7fff700 1 -- 172.21.15.116:0/89442079 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47d80180a0 con 0x7f47e015a5f0 2023-11-08T08:19:56.257 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:56.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47cc0682f0 msgr2=0x7f47cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47cc0682f0 0x7f47cc06a7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47d4005910 tx=0x7f47d4009040 comp rx=0 tx=0).stop 2023-11-08T08:19:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 msgr2=0x7f47e00cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 0x7f47e00cf170 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f47d800efc0 tx=0x7f47d800c3a0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 shutdown_connections 2023-11-08T08:19:56.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e015a5f0 0x7f47e00cf170 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47cc0682f0 0x7f47cc06a7a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e00b2ee0 0x7f47e0151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 --2- 172.21.15.116:0/89442079 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47e00cc200 0x7f47e00cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 >> 172.21.15.116:0/89442079 conn(0x7f47e006ea80 msgr2=0x7f47e00d0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 shutdown_connections 2023-11-08T08:19:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.257+0000 7f47e580e700 1 -- 172.21.15.116:0/89442079 wait complete. 2023-11-08T08:19:56.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.392+0000 7f2f74d0d700 1 Processor -- start 2023-11-08T08:19:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.393+0000 7f2f74d0d700 1 -- start start 2023-11-08T08:19:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.393+0000 7f2f74d0d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7015c8c0 0x7f2f70160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.393+0000 7f2f74d0d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f700b5dc0 0x7f2f7006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.393+0000 7f2f74d0d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700d2d30 con 0x7f2f7006b050 2023-11-08T08:19:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.393+0000 7f2f74d0d700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7006b4d0 con 0x7f2f700b5dc0 2023-11-08T08:19:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.393+0000 7f2f74d0d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700d0b40 con 0x7f2f7015c8c0 2023-11-08T08:19:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f700b5dc0 0x7f2f7006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f700b5dc0 0x7f2f7006ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35016/0 (socket says 172.21.15.116:35016) 2023-11-08T08:19:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/1907525138 learned_addr learned my addr 172.21.15.116:0/1907525138 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6e59c700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7015c8c0 0x7f2f70160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/1907525138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7015c8c0 msgr2=0x7f2f70160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7015c8c0 0x7f2f70160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/1907525138 >> v1:172.21.15.129:6789/0 conn(0x7f2f7006b050 legacy=0x7f2f700d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/1907525138 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f701611f0 con 0x7f2f700b5dc0 2023-11-08T08:19:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6e59c700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7015c8c0 0x7f2f70160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6dd9b700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7006ab10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2f600098f0 tx=0x7f2f60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9592cd435d5eea server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6d59a700 1 -- 172.21.15.116:0/1907525138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60004230 con 0x7f2f700b5dc0 2023-11-08T08:19:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6d59a700 1 -- 172.21.15.116:0/1907525138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60004390 con 0x7f2f700b5dc0 2023-11-08T08:19:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.394+0000 7f2f6d59a700 1 -- 172.21.15.116:0/1907525138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f600036a0 con 0x7f2f700b5dc0 2023-11-08T08:19:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 -- 172.21.15.116:0/1907525138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 msgr2=0x7f2f7006ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7006ab10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2f600098f0 tx=0x7f2f60009c00 comp rx=0 tx=0).stop 2023-11-08T08:19:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 -- 172.21.15.116:0/1907525138 shutdown_connections 2023-11-08T08:19:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7015c8c0 0x7f2f70160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/1907525138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7006ab10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 -- 172.21.15.116:0/1907525138 >> 172.21.15.116:0/1907525138 conn(0x7f2f7006c520 msgr2=0x7f2f7006c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 -- 172.21.15.116:0/1907525138 shutdown_connections 2023-11-08T08:19:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.395+0000 7f2f74d0d700 1 -- 172.21.15.116:0/1907525138 wait complete. 2023-11-08T08:19:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.396+0000 7f2f74d0d700 1 Processor -- start 2023-11-08T08:19:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.396+0000 7f2f74d0d700 1 -- start start 2023-11-08T08:19:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f74d0d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7006b050 0x7f2f700c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f74d0d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7015d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f74d0d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7015c8c0 0x7f2f7015f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f74d0d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700c7980 con 0x7f2f700b5dc0 2023-11-08T08:19:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f74d0d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700c9980 con 0x7f2f7015c8c0 2023-11-08T08:19:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f74d0d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700c83a0 con 0x7f2f7006b050 2023-11-08T08:19:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7015d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7015d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35020/0 (socket says 172.21.15.116:35020) 2023-11-08T08:19:56.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/2109697767 learned_addr learned my addr 172.21.15.116:0/2109697767 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:56.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/2109697767 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7006b050 msgr2=0x7f2f700c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7006b050 0x7f2f700c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/2109697767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7015c8c0 msgr2=0x7f2f7015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7015c8c0 0x7f2f7015f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 -- 172.21.15.116:0/2109697767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60009550 con 0x7f2f700b5dc0 2023-11-08T08:19:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f6dd9b700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7015d610 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009130 tx=0x7f2f60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.397+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60003e30 con 0x7f2f700b5dc0 2023-11-08T08:19:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.398+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6001f070 con 0x7f2f700b5dc0 2023-11-08T08:19:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.398+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f700cd760 con 0x7f2f700b5dc0 2023-11-08T08:19:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.398+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000b8a0 con 0x7f2f700b5dc0 2023-11-08T08:19:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.398+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f700d2a50 con 0x7f2f700b5dc0 2023-11-08T08:19:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.399+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f700cda40 con 0x7f2f700b5dc0 2023-11-08T08:19:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.399+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f6002a020 con 0x7f2f700b5dc0 2023-11-08T08:19:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.399+0000 7f2f5effd700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2f500682f0 0x7f2f5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.399+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2f600873a0 con 0x7f2f700b5dc0 2023-11-08T08:19:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.399+0000 7f2f6e59c700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2f500682f0 0x7f2f5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.400+0000 7f2f6e59c700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2f500682f0 0x7f2f5006a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2f64007920 tx=0x7f2f64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.404+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f60053c00 con 0x7f2f700b5dc0 2023-11-08T08:19:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.588+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f70067d70 con 0x7f2f700b5dc0 2023-11-08T08:19:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.589+0000 7f2f5effd700 1 -- 172.21.15.116:0/2109697767 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f60010380 con 0x7f2f700b5dc0 2023-11-08T08:19:56.590 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2f500682f0 msgr2=0x7f2f5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2f500682f0 0x7f2f5006a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2f64007920 tx=0x7f2f64008040 comp rx=0 tx=0).stop 2023-11-08T08:19:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 msgr2=0x7f2f7015d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7015d610 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009130 tx=0x7f2f60004750 comp rx=0 tx=0).stop 2023-11-08T08:19:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 shutdown_connections 2023-11-08T08:19:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2f500682f0 0x7f2f5006a7a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7015c8c0 0x7f2f7015f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2f7006b050 0x7f2f700c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 --2- 172.21.15.116:0/2109697767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f700b5dc0 0x7f2f7015d610 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 >> 172.21.15.116:0/2109697767 conn(0x7f2f7006c520 msgr2=0x7f2f700cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 shutdown_connections 2023-11-08T08:19:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.591+0000 7f2f74d0d700 1 -- 172.21.15.116:0/2109697767 wait complete. 2023-11-08T08:19:56.604 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.724+0000 7fc4204b5700 1 Processor -- start 2023-11-08T08:19:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.724+0000 7fc4204b5700 1 -- start start 2023-11-08T08:19:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc4204b5700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc4181599b0 0x7fc41815d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc4204b5700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4180af0a0 0x7fc41815a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc4204b5700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418159ec0 con 0x7fc4180b1650 2023-11-08T08:19:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc4204b5700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41815d8a0 con 0x7fc4180af0a0 2023-11-08T08:19:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc4204b5700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41815db80 con 0x7fc4181599b0 2023-11-08T08:19:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc41ea52700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4180af0a0 0x7fc41815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc41ea52700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4180af0a0 0x7fc41815a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35022/0 (socket says 172.21.15.116:35022) 2023-11-08T08:19:56.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc41e251700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc4180b1650 0x7fc4180b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52180/0 (socket says 172.21.15.116:52180) 2023-11-08T08:19:56.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.725+0000 7fc41ea52700 1 -- 172.21.15.116:0/42742223 learned_addr learned my addr 172.21.15.116:0/42742223 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:56.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41da50700 1 --2- 172.21.15.116:0/42742223 >> v2:172.21.15.129:3300/0 conn(0x7fc4181599b0 0x7fc41815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41ea52700 1 -- 172.21.15.116:0/42742223 >> v2:172.21.15.129:3300/0 conn(0x7fc4181599b0 msgr2=0x7fc41815d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:56.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41ea52700 1 --2- 172.21.15.116:0/42742223 >> v2:172.21.15.129:3300/0 conn(0x7fc4181599b0 0x7fc41815d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41ea52700 1 -- 172.21.15.116:0/42742223 >> v1:172.21.15.116:6789/0 conn(0x7fc4180b1650 legacy=0x7fc4180b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41ea52700 1 -- 172.21.15.116:0/42742223 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41815de60 con 0x7fc4180af0a0 2023-11-08T08:19:56.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41d24f700 1 -- 172.21.15.116:0/42742223 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125179190 0 0) 0x7fc418159ec0 con 0x7fc4180b1650 2023-11-08T08:19:56.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41ea52700 1 --2- 172.21.15.116:0/42742223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4180af0a0 0x7fc41815a350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc41806c730 tx=0x7fc41000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354752e1e6085547 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41d24f700 1 -- 172.21.15.116:0/42742223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410010070 con 0x7fc4180af0a0 2023-11-08T08:19:56.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41d24f700 1 -- 172.21.15.116:0/42742223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41000dd90 con 0x7fc4180af0a0 2023-11-08T08:19:56.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc41d24f700 1 -- 172.21.15.116:0/42742223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41000b310 con 0x7fc4180af0a0 2023-11-08T08:19:56.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc4204b5700 1 -- 172.21.15.116:0/42742223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4180af0a0 msgr2=0x7fc41815a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.726+0000 7fc4204b5700 1 --2- 172.21.15.116:0/42742223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4180af0a0 0x7fc41815a350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc41806c730 tx=0x7fc41000d430 comp rx=0 tx=0).stop 2023-11-08T08:19:56.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.727+0000 7fc4204b5700 1 -- 172.21.15.116:0/42742223 shutdown_connections 2023-11-08T08:19:56.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.727+0000 7fc4204b5700 1 --2- 172.21.15.116:0/42742223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4180af0a0 0x7fc41815a350 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.727+0000 7fc4204b5700 1 --2- 172.21.15.116:0/42742223 >> v2:172.21.15.129:3300/0 conn(0x7fc4181599b0 0x7fc41815d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.727+0000 7fc4204b5700 1 -- 172.21.15.116:0/42742223 >> 172.21.15.116:0/42742223 conn(0x7fc41806c010 msgr2=0x7fc4180cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:56.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.727+0000 7fc4204b5700 1 -- 172.21.15.116:0/42742223 shutdown_connections 2023-11-08T08:19:56.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.727+0000 7fc4204b5700 1 -- 172.21.15.116:0/42742223 wait complete. 2023-11-08T08:19:56.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 Processor -- start 2023-11-08T08:19:56.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 -- start start 2023-11-08T08:19:56.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4180af0a0 0x7fc4181539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc4180b1650 0x7fc418157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 0x7fc4181416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41813ec70 con 0x7fc418141240 2023-11-08T08:19:56.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc418141c00 con 0x7fc4180af0a0 2023-11-08T08:19:56.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc4204b5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41813ee80 con 0x7fc4180b1650 2023-11-08T08:19:56.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.728+0000 7fc41ea52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 0x7fc4181416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 0x7fc4181416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35038/0 (socket says 172.21.15.116:35038) 2023-11-08T08:19:56.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 -- 172.21.15.116:0/1390742689 learned_addr learned my addr 172.21.15.116:0/1390742689 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:56.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41da50700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc4180b1650 0x7fc418157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41e251700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4180af0a0 0x7fc4181539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 -- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc4180b1650 msgr2=0x7fc418157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc4180b1650 0x7fc418157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 -- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4180af0a0 msgr2=0x7fc4181539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4180af0a0 0x7fc4181539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 -- 172.21.15.116:0/1390742689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41000bd40 con 0x7fc418141240 2023-11-08T08:19:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc41ea52700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 0x7fc4181416c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc4180708b0 tx=0x7fc410004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41000bbc0 con 0x7fc418141240 2023-11-08T08:19:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41000b4c0 con 0x7fc418141240 2023-11-08T08:19:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41815a890 con 0x7fc418141240 2023-11-08T08:19:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410004040 con 0x7fc418141240 2023-11-08T08:19:56.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.729+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41815d590 con 0x7fc418141240 2023-11-08T08:19:56.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.730+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41815aa20 con 0x7fc418141240 2023-11-08T08:19:56.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.730+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc41002d070 con 0x7fc418141240 2023-11-08T08:19:56.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.731+0000 7fc40effd700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3fc0682f0 0x7fc3fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.731+0000 7fc41e251700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3fc0682f0 0x7fc3fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.731+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc4100894b0 con 0x7fc418141240 2023-11-08T08:19:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.731+0000 7fc41e251700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3fc0682f0 0x7fc3fc06a7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc408000f60 tx=0x7fc408009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.735+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc410055d10 con 0x7fc418141240 2023-11-08T08:19:56.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.934+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc4180743f0 con 0x7fc418141240 2023-11-08T08:19:56.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.934+0000 7fc40effd700 1 -- 172.21.15.116:0/1390742689 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v32) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc410059330 con 0x7fc418141240 2023-11-08T08:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3fc0682f0 msgr2=0x7fc3fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3fc0682f0 0x7fc3fc06a7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc408000f60 tx=0x7fc408009040 comp rx=0 tx=0).stop 2023-11-08T08:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 msgr2=0x7fc4181416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 0x7fc4181416c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc4180708b0 tx=0x7fc410004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 shutdown_connections 2023-11-08T08:19:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc418141240 0x7fc4181416c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3fc0682f0 0x7fc3fc06a7a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4180af0a0 0x7fc4181539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 --2- 172.21.15.116:0/1390742689 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc4180b1650 0x7fc418157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 >> 172.21.15.116:0/1390742689 conn(0x7fc41806c010 msgr2=0x7fc4180b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 shutdown_connections 2023-11-08T08:19:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:56.936+0000 7fc4204b5700 1 -- 172.21.15.116:0/1390742689 wait complete. 2023-11-08T08:19:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:56.948 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:19:56.949 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:19:56.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:19:57.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.127+0000 7f982f5a1700 1 Processor -- start 2023-11-08T08:19:57.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.127+0000 7f982f5a1700 1 -- start start 2023-11-08T08:19:57.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982f5a1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 0x7f9828072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982f5a1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 0x7f98280ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982f5a1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98280726c0 con 0x7f98280b0880 2023-11-08T08:19:57.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982f5a1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982806bc50 con 0x7f982806b7b0 2023-11-08T08:19:57.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982f5a1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98280b43f0 con 0x7f98280ad1f0 2023-11-08T08:19:57.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982db3e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 0x7f98280ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982cb3c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 0x7f9828072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982db3e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 0x7f98280ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:47298/0 (socket says 172.21.15.116:47298) 2023-11-08T08:19:57.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982db3e700 1 -- 172.21.15.116:0/1244115950 learned_addr learned my addr 172.21.15.116:0/1244115950 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:57.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982cb3c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 0x7f9828072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:55018/0 (socket says 172.21.15.116:55018) 2023-11-08T08:19:57.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.128+0000 7f982d33d700 1 --1- 172.21.15.116:0/1244115950 >> v1:172.21.15.129:6789/0 conn(0x7f98280b0880 0x7f98280aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:38530/0 (socket says 172.21.15.116:38530) 2023-11-08T08:19:57.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982db3e700 1 -- 172.21.15.116:0/1244115950 >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 msgr2=0x7f9828072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:57.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982db3e700 1 --2- 172.21.15.116:0/1244115950 >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 0x7f9828072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982db3e700 1 -- 172.21.15.116:0/1244115950 >> v1:172.21.15.129:6789/0 conn(0x7f98280b0880 legacy=0x7f98280aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982db3e700 1 -- 172.21.15.116:0/1244115950 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98280adb90 con 0x7f98280ad1f0 2023-11-08T08:19:57.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982cb3c700 1 --2- 172.21.15.116:0/1244115950 >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 0x7f9828072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:57.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982db3e700 1 --2- 172.21.15.116:0/1244115950 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 0x7f98280ad650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9820009e50 tx=0x7f982000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3961e16daca731f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f981ffff700 1 -- 172.21.15.116:0/1244115950 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000e070 con 0x7f98280ad1f0 2023-11-08T08:19:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f981ffff700 1 -- 172.21.15.116:0/1244115950 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982000b910 con 0x7f98280ad1f0 2023-11-08T08:19:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f981ffff700 1 -- 172.21.15.116:0/1244115950 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000bcc0 con 0x7f98280ad1f0 2023-11-08T08:19:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982f5a1700 1 -- 172.21.15.116:0/1244115950 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 msgr2=0x7f98280ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.129+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1244115950 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 0x7f98280ad650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9820009e50 tx=0x7f982000b330 comp rx=0 tx=0).stop 2023-11-08T08:19:57.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.130+0000 7f982f5a1700 1 -- 172.21.15.116:0/1244115950 shutdown_connections 2023-11-08T08:19:57.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.130+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1244115950 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280ad1f0 0x7f98280ad650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.130+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1244115950 >> v2:172.21.15.129:3300/0 conn(0x7f982806b7b0 0x7f9828072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.130+0000 7f982f5a1700 1 -- 172.21.15.116:0/1244115950 >> 172.21.15.116:0/1244115950 conn(0x7f98280b2fc0 msgr2=0x7f9828071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:57.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.130+0000 7f982f5a1700 1 -- 172.21.15.116:0/1244115950 shutdown_connections 2023-11-08T08:19:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.130+0000 7f982f5a1700 1 -- 172.21.15.116:0/1244115950 wait complete. 2023-11-08T08:19:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 Processor -- start 2023-11-08T08:19:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 -- start start 2023-11-08T08:19:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 0x7f9828159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98280ad1f0 0x7f982809c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280b0880 0x7f982809c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982815a410 con 0x7f982806b7b0 2023-11-08T08:19:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982809a630 con 0x7f98280ad1f0 2023-11-08T08:19:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.131+0000 7f982f5a1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9828099a20 con 0x7f98280b0880 2023-11-08T08:19:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 0x7f9828159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 0x7f9828159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35040/0 (socket says 172.21.15.116:35040) 2023-11-08T08:19:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 -- 172.21.15.116:0/1962048478 learned_addr learned my addr 172.21.15.116:0/1962048478 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982cb3c700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98280ad1f0 0x7f982809c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982db3e700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280b0880 0x7f982809c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 -- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280b0880 msgr2=0x7f982809c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280b0880 0x7f982809c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 -- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98280ad1f0 msgr2=0x7f982809c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98280ad1f0 0x7f982809c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 -- 172.21.15.116:0/1962048478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240094a0 con 0x7f982806b7b0 2023-11-08T08:19:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982d33d700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 0x7f9828159ed0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f981800c520 tx=0x7f981800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818005cb0 con 0x7f982806b7b0 2023-11-08T08:19:57.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818005e10 con 0x7f982806b7b0 2023-11-08T08:19:57.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180072d0 con 0x7f982806b7b0 2023-11-08T08:19:57.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9820009b30 con 0x7f982806b7b0 2023-11-08T08:19:57.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.132+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98280b4d00 con 0x7f982806b7b0 2023-11-08T08:19:57.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.133+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98280b46d0 con 0x7f982806b7b0 2023-11-08T08:19:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.133+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9818019030 con 0x7f982806b7b0 2023-11-08T08:19:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.134+0000 7f981dffb700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f980c0682f0 0x7f980c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.134+0000 7f982cb3c700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f980c0682f0 0x7f980c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.134+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9818084db0 con 0x7f982806b7b0 2023-11-08T08:19:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.134+0000 7f982cb3c700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f980c0682f0 0x7f980c06a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f982806e100 tx=0x7f9824009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.138+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f981804b5e0 con 0x7f982806b7b0 2023-11-08T08:19:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.318+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98280b4fe0 con 0x7f982806b7b0 2023-11-08T08:19:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.319+0000 7f981dffb700 1 -- 172.21.15.116:0/1962048478 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f9818017020 con 0x7f982806b7b0 2023-11-08T08:19:57.321 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:19:57.322 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:46.066442+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:19:57.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f980c0682f0 msgr2=0x7f980c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f980c0682f0 0x7f980c06a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f982806e100 tx=0x7f9824009210 comp rx=0 tx=0).stop 2023-11-08T08:19:57.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 msgr2=0x7f9828159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 0x7f9828159ed0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f981800c520 tx=0x7f981800c830 comp rx=0 tx=0).stop 2023-11-08T08:19:57.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 shutdown_connections 2023-11-08T08:19:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f98280b0880 0x7f982809c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f982806b7b0 0x7f9828159ed0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f980c0682f0 0x7f980c06a7a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 --2- 172.21.15.116:0/1962048478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98280ad1f0 0x7f982809c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.321+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 >> 172.21.15.116:0/1962048478 conn(0x7f98280b2fc0 msgr2=0x7f982806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:57.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.322+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 shutdown_connections 2023-11-08T08:19:57.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.322+0000 7f982f5a1700 1 -- 172.21.15.116:0/1962048478 wait complete. 2023-11-08T08:19:57.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:19:43.321551+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:19:44.023597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:19:57.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:19:57.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.458+0000 7f30e7ca6700 1 Processor -- start 2023-11-08T08:19:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e7ca6700 1 -- start start 2023-11-08T08:19:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e7ca6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e015fe50 0x7f30e006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e7ca6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f30e00b3b80 0x7f30e0160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e7ca6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e0161370 con 0x7f30e00b27d0 2023-11-08T08:19:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e7ca6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e0160220 con 0x7f30e00b3b80 2023-11-08T08:19:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e7ca6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00cf700 con 0x7f30e015fe50 2023-11-08T08:19:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e6243700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f30e00b27d0 0x7f30e00cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52182/0 (socket says 172.21.15.116:52182) 2023-11-08T08:19:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.459+0000 7f30e6243700 1 -- 172.21.15.116:0/1621462646 learned_addr learned my addr 172.21.15.116:0/1621462646 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5a42700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e015fe50 0x7f30e006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5241700 1 --2- 172.21.15.116:0/1621462646 >> v2:172.21.15.129:3300/0 conn(0x7f30e00b3b80 0x7f30e0160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e4a40700 1 -- 172.21.15.116:0/1621462646 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382114832 0 0) 0x7f30e0161370 con 0x7f30e00b27d0 2023-11-08T08:19:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e4a40700 1 -- 172.21.15.116:0/1621462646 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc0035b0 con 0x7f30e00b27d0 2023-11-08T08:19:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e4a40700 1 -- 172.21.15.116:0/1621462646 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608650527 0 0) 0x7f30cc0035b0 con 0x7f30e00b27d0 2023-11-08T08:19:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5241700 1 -- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e015fe50 msgr2=0x7f30e006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5241700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e015fe50 0x7f30e006d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5241700 1 -- 172.21.15.116:0/1621462646 >> v1:172.21.15.116:6789/0 conn(0x7f30e00b27d0 legacy=0x7f30e00cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5241700 1 -- 172.21.15.116:0/1621462646 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00b3fd0 con 0x7f30e00b3b80 2023-11-08T08:19:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5a42700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e015fe50 0x7f30e006d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e5241700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e0160ad0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009840 tx=0x7f30d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=122e218bdc80aac5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e4a40700 1 -- 172.21.15.116:0/1621462646 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80019f0 con 0x7f30e00b3b80 2023-11-08T08:19:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e4a40700 1 -- 172.21.15.116:0/1621462646 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d4004030 con 0x7f30e00b3b80 2023-11-08T08:19:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.460+0000 7f30e4a40700 1 -- 172.21.15.116:0/1621462646 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d40043c0 con 0x7f30e00b3b80 2023-11-08T08:19:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 msgr2=0x7f30e0160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e0160ad0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009840 tx=0x7f30d4009b50 comp rx=0 tx=0).stop 2023-11-08T08:19:57.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1621462646 shutdown_connections 2023-11-08T08:19:57.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e015fe50 0x7f30e006d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1621462646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e0160ad0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1621462646 >> 172.21.15.116:0/1621462646 conn(0x7f30e006ea80 msgr2=0x7f30e006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1621462646 shutdown_connections 2023-11-08T08:19:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.461+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1621462646 wait complete. 2023-11-08T08:19:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.462+0000 7f30e7ca6700 1 Processor -- start 2023-11-08T08:19:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.462+0000 7f30e7ca6700 1 -- start start 2023-11-08T08:19:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e7ca6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e00b27d0 0x7f30e0154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e7ca6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e00b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e7ca6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 0x7f30e00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e7ca6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0155270 con 0x7f30e015fe50 2023-11-08T08:19:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e7ca6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00b5370 con 0x7f30e00b3b80 2023-11-08T08:19:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e7ca6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00ca410 con 0x7f30e00b27d0 2023-11-08T08:19:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 0x7f30e00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 0x7f30e00b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35048/0 (socket says 172.21.15.116:35048) 2023-11-08T08:19:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 -- 172.21.15.116:0/1650993700 learned_addr learned my addr 172.21.15.116:0/1650993700 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e5a42700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e00b27d0 0x7f30e0154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e5241700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e00b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 -- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e00b27d0 msgr2=0x7f30e0154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e00b27d0 0x7f30e0154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 -- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 msgr2=0x7f30e00b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e00b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.463+0000 7f30e6243700 1 -- 172.21.15.116:0/1650993700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc0094a0 con 0x7f30e015fe50 2023-11-08T08:19:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.464+0000 7f30e6243700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 0x7f30e00b0cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f30d800cc30 tx=0x7f30d800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.464+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d800aac0 con 0x7f30e015fe50 2023-11-08T08:19:57.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.464+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d800ac40 con 0x7f30e015fe50 2023-11-08T08:19:57.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.464+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d40094a0 con 0x7f30e015fe50 2023-11-08T08:19:57.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.464+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80070d0 con 0x7f30e015fe50 2023-11-08T08:19:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.464+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e00cf0b0 con 0x7f30e015fe50 2023-11-08T08:19:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.465+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00cedd0 con 0x7f30e015fe50 2023-11-08T08:19:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.465+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30d801a030 con 0x7f30e015fe50 2023-11-08T08:19:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.465+0000 7f30d27fc700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30c40682f0 0x7f30c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.466+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f30d8085910 con 0x7f30e015fe50 2023-11-08T08:19:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.466+0000 7f30e5a42700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30c40682f0 0x7f30c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.466+0000 7f30e5a42700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30c40682f0 0x7f30c406a7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009cd0 tx=0x7f30dc009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.470+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30d8052170 con 0x7f30e015fe50 2023-11-08T08:19:57.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.651+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f30e00cf390 con 0x7f30e015fe50 2023-11-08T08:19:57.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.651+0000 7f30d27fc700 1 -- 172.21.15.116:0/1650993700 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7f30d80180a0 con 0x7f30e015fe50 2023-11-08T08:19:57.652 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:19:57.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30c40682f0 msgr2=0x7f30c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30c40682f0 0x7f30c406a7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009cd0 tx=0x7f30dc009210 comp rx=0 tx=0).stop 2023-11-08T08:19:57.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 msgr2=0x7f30e00b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 0x7f30e00b0cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f30d800cc30 tx=0x7f30d800cf40 comp rx=0 tx=0).stop 2023-11-08T08:19:57.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 shutdown_connections 2023-11-08T08:19:57.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e015fe50 0x7f30e00b0cf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f30e00b27d0 0x7f30e0154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30c40682f0 0x7f30c406a7a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 --2- 172.21.15.116:0/1650993700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30e00b3b80 0x7f30e00b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 >> 172.21.15.116:0/1650993700 conn(0x7f30e006ea80 msgr2=0x7f30e015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:57.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 shutdown_connections 2023-11-08T08:19:57.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.653+0000 7f30e7ca6700 1 -- 172.21.15.116:0/1650993700 wait complete. 2023-11-08T08:19:57.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:19:57.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.788+0000 7f3455605700 1 Processor -- start 2023-11-08T08:19:57.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.789+0000 7f3455605700 1 -- start start 2023-11-08T08:19:57.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.789+0000 7f3455605700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f345006a980 0x7f34500ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.789+0000 7f3455605700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f3450071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.789+0000 7f3455605700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345015cc10 con 0x7f34500cf710 2023-11-08T08:19:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.789+0000 7f3455605700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345006c9b0 con 0x7f345006a980 2023-11-08T08:19:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.789+0000 7f3455605700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345006cc90 con 0x7f3450071890 2023-11-08T08:19:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f3450071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f3450071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:47338/0 (socket says 172.21.15.116:47338) 2023-11-08T08:19:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 -- 172.21.15.116:0/1024610451 learned_addr learned my addr 172.21.15.116:0/1024610451 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344e7fc700 1 --2- 172.21.15.116:0/1024610451 >> v2:172.21.15.129:3300/0 conn(0x7f345006a980 0x7f34500ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 -- 172.21.15.116:0/1024610451 >> v2:172.21.15.129:3300/0 conn(0x7f345006a980 msgr2=0x7f34500ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 --2- 172.21.15.116:0/1024610451 >> v2:172.21.15.129:3300/0 conn(0x7f345006a980 0x7f34500ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 -- 172.21.15.116:0/1024610451 >> v1:172.21.15.129:6789/0 conn(0x7f34500cf710 legacy=0x7f345006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 -- 172.21.15.116:0/1024610451 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345006cf70 con 0x7f3450071890 2023-11-08T08:19:57.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344e7fc700 1 --2- 172.21.15.116:0/1024610451 >> v2:172.21.15.129:3300/0 conn(0x7f345006a980 0x7f34500ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:19:57.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344f7fe700 1 --2- 172.21.15.116:0/1024610451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f3450071220 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3448009e50 tx=0x7f344800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdda58bc9c8a3429 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344dffb700 1 -- 172.21.15.116:0/1024610451 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800e070 con 0x7f3450071890 2023-11-08T08:19:57.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344dffb700 1 -- 172.21.15.116:0/1024610451 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344800b910 con 0x7f3450071890 2023-11-08T08:19:57.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.790+0000 7f344dffb700 1 -- 172.21.15.116:0/1024610451 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800bcc0 con 0x7f3450071890 2023-11-08T08:19:57.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 -- 172.21.15.116:0/1024610451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 msgr2=0x7f3450071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 --2- 172.21.15.116:0/1024610451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f3450071220 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3448009e50 tx=0x7f344800b330 comp rx=0 tx=0).stop 2023-11-08T08:19:57.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 -- 172.21.15.116:0/1024610451 shutdown_connections 2023-11-08T08:19:57.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 --2- 172.21.15.116:0/1024610451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f3450071220 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 --2- 172.21.15.116:0/1024610451 >> v2:172.21.15.129:3300/0 conn(0x7f345006a980 0x7f34500ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 -- 172.21.15.116:0/1024610451 >> 172.21.15.116:0/1024610451 conn(0x7f345006c010 msgr2=0x7f34500cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:57.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 -- 172.21.15.116:0/1024610451 shutdown_connections 2023-11-08T08:19:57.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.791+0000 7f3455605700 1 -- 172.21.15.116:0/1024610451 wait complete. 2023-11-08T08:19:57.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.792+0000 7f3455605700 1 Processor -- start 2023-11-08T08:19:57.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 -- start start 2023-11-08T08:19:57.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f345006a980 0x7f3450148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f34500b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 0x7f34500b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450149520 con 0x7f34500cf710 2023-11-08T08:19:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345015dd80 con 0x7f345006a980 2023-11-08T08:19:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.793+0000 7f3455605700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345015e060 con 0x7f3450071890 2023-11-08T08:19:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 0x7f34500b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 0x7f34500b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35056/0 (socket says 172.21.15.116:35056) 2023-11-08T08:19:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 -- 172.21.15.116:0/1423816134 learned_addr learned my addr 172.21.15.116:0/1423816134 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344effd700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f345006a980 0x7f3450148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344e7fc700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f34500b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 -- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 msgr2=0x7f34500b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f34500b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 -- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f345006a980 msgr2=0x7f3450148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f345006a980 0x7f3450148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 -- 172.21.15.116:0/1423816134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400094a0 con 0x7f34500cf710 2023-11-08T08:19:57.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344effd700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f345006a980 0x7f3450148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:57.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f344f7fe700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 0x7f34500b4920 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f345006cdd0 tx=0x7f3448004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448004960 con 0x7f34500cf710 2023-11-08T08:19:57.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34480099b0 con 0x7f34500cf710 2023-11-08T08:19:57.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344801c680 con 0x7f34500cf710 2023-11-08T08:19:57.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3448009b30 con 0x7f34500cf710 2023-11-08T08:19:57.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.794+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345006d530 con 0x7f34500cf710 2023-11-08T08:19:57.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.795+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345006d250 con 0x7f34500cf710 2023-11-08T08:19:57.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.796+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f344802c070 con 0x7f34500cf710 2023-11-08T08:19:57.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.796+0000 7f34377fe700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f343c0682f0 0x7f343c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:57.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.796+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3448087e70 con 0x7f34500cf710 2023-11-08T08:19:57.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.796+0000 7f344effd700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f343c0682f0 0x7f343c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:57.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.797+0000 7f344effd700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f343c0682f0 0x7f343c06a7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3444000f60 tx=0x7f3444009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:57.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.800+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3448054770 con 0x7f34500cf710 2023-11-08T08:19:57.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.976+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3450160480 con 0x7f34500cf710 2023-11-08T08:19:57.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.977+0000 7f34377fe700 1 -- 172.21.15.116:0/1423816134 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3448057d90 con 0x7f34500cf710 2023-11-08T08:19:57.978 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:19:57.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f343c0682f0 msgr2=0x7f343c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f343c0682f0 0x7f343c06a7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3444000f60 tx=0x7f3444009040 comp rx=0 tx=0).stop 2023-11-08T08:19:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 msgr2=0x7f34500b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 0x7f34500b4920 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f345006cdd0 tx=0x7f3448004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 shutdown_connections 2023-11-08T08:19:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34500cf710 0x7f34500b4920 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f343c0682f0 0x7f343c06a7a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f345006a980 0x7f3450148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 --2- 172.21.15.116:0/1423816134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3450071890 0x7f34500b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 >> 172.21.15.116:0/1423816134 conn(0x7f345006c010 msgr2=0x7f34500b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 shutdown_connections 2023-11-08T08:19:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:57.979+0000 7f3455605700 1 -- 172.21.15.116:0/1423816134 wait complete. 2023-11-08T08:19:57.992 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:19:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.114+0000 7fc950997700 1 Processor -- start 2023-11-08T08:19:58.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.114+0000 7fc950997700 1 -- start start 2023-11-08T08:19:58.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc950997700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc94c0cf710 0x7fc94c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc950997700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c15cc10 con 0x7fc94c06a980 2023-11-08T08:19:58.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc950997700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c06c9b0 con 0x7fc94c071890 2023-11-08T08:19:58.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc950997700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c06cc90 con 0x7fc94c0cf710 2023-11-08T08:19:58.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94a59c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc94c0cf710 0x7fc94c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94ad9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc94c06a980 0x7fc94c160480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52192/0 (socket says 172.21.15.116:52192) 2023-11-08T08:19:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94ad9d700 1 -- 172.21.15.116:0/4211101019 learned_addr learned my addr 172.21.15.116:0/4211101019 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94a59c700 1 -- 172.21.15.116:0/4211101019 >> v1:172.21.15.129:6789/0 conn(0x7fc94c071890 legacy=0x7fc94c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94a59c700 1 -- 172.21.15.116:0/4211101019 >> v1:172.21.15.116:6789/0 conn(0x7fc94c06a980 legacy=0x7fc94c160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94959a700 1 -- 172.21.15.116:0/4211101019 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127680403 0 0) 0x7fc94c15cc10 con 0x7fc94c06a980 2023-11-08T08:19:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94a59c700 1 -- 172.21.15.116:0/4211101019 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c06cf70 con 0x7fc94c0cf710 2023-11-08T08:19:58.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94a59c700 1 --2- 172.21.15.116:0/4211101019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0cf710 0x7fc94c0ce740 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009870 tx=0x7fc93c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb432a702037ef6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94959a700 1 -- 172.21.15.116:0/4211101019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c004230 con 0x7fc94c0cf710 2023-11-08T08:19:58.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94959a700 1 -- 172.21.15.116:0/4211101019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c004390 con 0x7fc94c0cf710 2023-11-08T08:19:58.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.115+0000 7fc94959a700 1 -- 172.21.15.116:0/4211101019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0036a0 con 0x7fc94c0cf710 2023-11-08T08:19:58.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 -- 172.21.15.116:0/4211101019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0cf710 msgr2=0x7fc94c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 --2- 172.21.15.116:0/4211101019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0cf710 0x7fc94c0ce740 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009870 tx=0x7fc93c009b80 comp rx=0 tx=0).stop 2023-11-08T08:19:58.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 -- 172.21.15.116:0/4211101019 shutdown_connections 2023-11-08T08:19:58.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 --2- 172.21.15.116:0/4211101019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0cf710 0x7fc94c0ce740 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 -- 172.21.15.116:0/4211101019 >> 172.21.15.116:0/4211101019 conn(0x7fc94c06c010 msgr2=0x7fc94c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:58.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 -- 172.21.15.116:0/4211101019 shutdown_connections 2023-11-08T08:19:58.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.116+0000 7fc950997700 1 -- 172.21.15.116:0/4211101019 wait complete. 2023-11-08T08:19:58.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.117+0000 7fc950997700 1 Processor -- start 2023-11-08T08:19:58.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.117+0000 7fc950997700 1 -- start start 2023-11-08T08:19:58.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc950997700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c06a980 0x7fc94c150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc950997700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 0x7fc94c14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc950997700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc94c0cf710 0x7fc94c15e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc950997700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c14ff30 con 0x7fc94c071890 2023-11-08T08:19:58.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc950997700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c152c00 con 0x7fc94c06a980 2023-11-08T08:19:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc950997700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c152ee0 con 0x7fc94c0cf710 2023-11-08T08:19:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 0x7fc94c14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 0x7fc94c14f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35070/0 (socket says 172.21.15.116:35070) 2023-11-08T08:19:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 -- 172.21.15.116:0/2651941424 learned_addr learned my addr 172.21.15.116:0/2651941424 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 -- 172.21.15.116:0/2651941424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc94c0cf710 msgr2=0x7fc94c15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc94c0cf710 0x7fc94c15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 -- 172.21.15.116:0/2651941424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c06a980 msgr2=0x7fc94c150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.118+0000 7fc949d9b700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c06a980 0x7fc94c150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc949d9b700 1 -- 172.21.15.116:0/2651941424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c0094a0 con 0x7fc94c071890 2023-11-08T08:19:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc949d9b700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 0x7fc94c14f9f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc94000c4d0 tx=0x7fc94000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940011790 con 0x7fc94c071890 2023-11-08T08:19:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9400118f0 con 0x7fc94c071890 2023-11-08T08:19:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c06d530 con 0x7fc94c071890 2023-11-08T08:19:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940011ca0 con 0x7fc94c071890 2023-11-08T08:19:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.119+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c06d250 con 0x7fc94c071890 2023-11-08T08:19:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.120+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c0743f0 con 0x7fc94c071890 2023-11-08T08:19:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.120+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc940010050 con 0x7fc94c071890 2023-11-08T08:19:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.121+0000 7fc93affd700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc92c0682f0 0x7fc92c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.121+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc940085a10 con 0x7fc94c071890 2023-11-08T08:19:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.121+0000 7fc94a59c700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc92c0682f0 0x7fc92c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.121+0000 7fc94a59c700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc92c0682f0 0x7fc92c06a7a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0090b0 tx=0x7fc93c013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.125+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc940052270 con 0x7fc94c071890 2023-11-08T08:19:58.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.302+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc94c0711a0 con 0x7fc94c071890 2023-11-08T08:19:58.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.302+0000 7fc93affd700 1 -- 172.21.15.116:0/2651941424 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc940055890 con 0x7fc94c071890 2023-11-08T08:19:58.304 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:19:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc92c0682f0 msgr2=0x7fc92c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc92c0682f0 0x7fc92c06a7a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0090b0 tx=0x7fc93c013040 comp rx=0 tx=0).stop 2023-11-08T08:19:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 msgr2=0x7fc94c14f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 0x7fc94c14f9f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc94000c4d0 tx=0x7fc94000c890 comp rx=0 tx=0).stop 2023-11-08T08:19:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 shutdown_connections 2023-11-08T08:19:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc94c0cf710 0x7fc94c15e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc92c0682f0 0x7fc92c06a7a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c06a980 0x7fc94c150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 --2- 172.21.15.116:0/2651941424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c071890 0x7fc94c14f9f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 >> 172.21.15.116:0/2651941424 conn(0x7fc94c06c010 msgr2=0x7fc94c0adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 shutdown_connections 2023-11-08T08:19:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.305+0000 7fc950997700 1 -- 172.21.15.116:0/2651941424 wait complete. 2023-11-08T08:19:58.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:19:58.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.438+0000 7f2363ead700 1 Processor -- start 2023-11-08T08:19:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.438+0000 7f2363ead700 1 -- start start 2023-11-08T08:19:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2363ead700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c15c810 0x7f235c160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2363ead700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f235c06a270 0x7f235c06a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2363ead700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c06ac10 con 0x7f235c0718e0 2023-11-08T08:19:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2363ead700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235c15cbe0 con 0x7f235c06a270 2023-11-08T08:19:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2363ead700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235c1607a0 con 0x7f235c15c810 2023-11-08T08:19:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f236244a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f235c06a270 0x7f235c06a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2361448700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f235c0718e0 0x7f235c06c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52208/0 (socket says 172.21.15.116:52208) 2023-11-08T08:19:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2361448700 1 -- 172.21.15.116:0/2693763326 learned_addr learned my addr 172.21.15.116:0/2693763326 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.439+0000 7f2361c49700 1 --2- 172.21.15.116:0/2693763326 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c15c810 0x7f235c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f236244a700 1 -- 172.21.15.116:0/2693763326 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c15c810 msgr2=0x7f235c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:19:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f236244a700 1 --2- 172.21.15.116:0/2693763326 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c15c810 0x7f235c160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f236244a700 1 -- 172.21.15.116:0/2693763326 >> v1:172.21.15.116:6789/0 conn(0x7f235c0718e0 legacy=0x7f235c06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f236244a700 1 -- 172.21.15.116:0/2693763326 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c160a50 con 0x7f235c06a270 2023-11-08T08:19:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2360c47700 1 -- 172.21.15.116:0/2693763326 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249393273 0 0) 0x7f235c06ac10 con 0x7f235c0718e0 2023-11-08T08:19:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f236244a700 1 --2- 172.21.15.116:0/2693763326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c06a270 0x7f235c06a6d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2354009d90 tx=0x7f235400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407db722f910abd server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2360c47700 1 -- 172.21.15.116:0/2693763326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235400e070 con 0x7f235c06a270 2023-11-08T08:19:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2360c47700 1 -- 172.21.15.116:0/2693763326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235400bbd0 con 0x7f235c06a270 2023-11-08T08:19:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2360c47700 1 -- 172.21.15.116:0/2693763326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354012430 con 0x7f235c06a270 2023-11-08T08:19:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2363ead700 1 -- 172.21.15.116:0/2693763326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c06a270 msgr2=0x7f235c06a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2363ead700 1 --2- 172.21.15.116:0/2693763326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c06a270 0x7f235c06a6d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2354009d90 tx=0x7f235400b330 comp rx=0 tx=0).stop 2023-11-08T08:19:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2363ead700 1 -- 172.21.15.116:0/2693763326 shutdown_connections 2023-11-08T08:19:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2363ead700 1 --2- 172.21.15.116:0/2693763326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c06a270 0x7f235c06a6d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2363ead700 1 --2- 172.21.15.116:0/2693763326 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c15c810 0x7f235c160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.440+0000 7f2363ead700 1 -- 172.21.15.116:0/2693763326 >> 172.21.15.116:0/2693763326 conn(0x7f235c06c010 msgr2=0x7f235c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:58.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.441+0000 7f2363ead700 1 -- 172.21.15.116:0/2693763326 shutdown_connections 2023-11-08T08:19:58.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.441+0000 7f2363ead700 1 -- 172.21.15.116:0/2693763326 wait complete. 2023-11-08T08:19:58.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 Processor -- start 2023-11-08T08:19:58.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 -- start start 2023-11-08T08:19:58.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c06a270 0x7f235c0c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f235c0718e0 0x7f235c06da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 0x7f235c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c144e90 con 0x7f235c15c810 2023-11-08T08:19:58.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c145170 con 0x7f235c0718e0 2023-11-08T08:19:58.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.442+0000 7f2363ead700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235c145450 con 0x7f235c06a270 2023-11-08T08:19:58.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 0x7f235c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 0x7f235c06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35084/0 (socket says 172.21.15.116:35084) 2023-11-08T08:19:58.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 -- 172.21.15.116:0/3944869575 learned_addr learned my addr 172.21.15.116:0/3944869575 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:58.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f2361448700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f235c0718e0 0x7f235c06da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f2361c49700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c06a270 0x7f235c0c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 -- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c06a270 msgr2=0x7f235c0c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c06a270 0x7f235c0c3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 -- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f235c0718e0 msgr2=0x7f235c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f235c0718e0 0x7f235c06da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 -- 172.21.15.116:0/3944869575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354009a70 con 0x7f235c15c810 2023-11-08T08:19:58.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f236244a700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 0x7f235c06df40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f235c067450 tx=0x7f2354004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354012430 con 0x7f235c15c810 2023-11-08T08:19:58.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2354020070 con 0x7f235c15c810 2023-11-08T08:19:58.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235c161010 con 0x7f235c15c810 2023-11-08T08:19:58.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354004280 con 0x7f235c15c810 2023-11-08T08:19:58.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.443+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235c160d30 con 0x7f235c15c810 2023-11-08T08:19:58.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.444+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235c0ceb20 con 0x7f235c15c810 2023-11-08T08:19:58.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.445+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f23540043e0 con 0x7f235c15c810 2023-11-08T08:19:58.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.445+0000 7f23527fc700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f23400682f0 0x7f234006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.445+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2354022070 con 0x7f235c15c810 2023-11-08T08:19:58.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.445+0000 7f2361c49700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f23400682f0 0x7f234006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.446+0000 7f2361c49700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f23400682f0 0x7f234006a7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f234c006b20 tx=0x7f234c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.449+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2354054c40 con 0x7f235c15c810 2023-11-08T08:19:58.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.628+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f235c145910 con 0x7f235c15c810 2023-11-08T08:19:58.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.628+0000 7f23527fc700 1 -- 172.21.15.116:0/3944869575 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2354058260 con 0x7f235c15c810 2023-11-08T08:19:58.630 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:19:58.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f23400682f0 msgr2=0x7f234006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f23400682f0 0x7f234006a7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f234c006b20 tx=0x7f234c008040 comp rx=0 tx=0).stop 2023-11-08T08:19:58.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 msgr2=0x7f235c06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 0x7f235c06df40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f235c067450 tx=0x7f2354004ad0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 shutdown_connections 2023-11-08T08:19:58.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c15c810 0x7f235c06df40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f235c06a270 0x7f235c0c3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f23400682f0 0x7f234006a7a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 --2- 172.21.15.116:0/3944869575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f235c0718e0 0x7f235c06da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 >> 172.21.15.116:0/3944869575 conn(0x7f235c06c010 msgr2=0x7f235c0b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:58.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 shutdown_connections 2023-11-08T08:19:58.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.630+0000 7f2363ead700 1 -- 172.21.15.116:0/3944869575 wait complete. 2023-11-08T08:19:58.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:19:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.763+0000 7f358cb7a700 1 Processor -- start 2023-11-08T08:19:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.763+0000 7f358cb7a700 1 -- start start 2023-11-08T08:19:58.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.763+0000 7f358cb7a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35881599b0 0x7f358815d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.763+0000 7f358cb7a700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3588159ec0 con 0x7f35880af0a0 2023-11-08T08:19:58.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.763+0000 7f358cb7a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358815aa60 con 0x7f35880b1650 2023-11-08T08:19:58.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.763+0000 7f358cb7a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358815d8a0 con 0x7f35881599b0 2023-11-08T08:19:58.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3586d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f35880af0a0 0x7f358815a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52210/0 (socket says 172.21.15.116:52210) 2023-11-08T08:19:58.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3586d9d700 1 -- 172.21.15.116:0/1394105721 learned_addr learned my addr 172.21.15.116:0/1394105721 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:19:58.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3585d9b700 1 --2- 172.21.15.116:0/1394105721 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35881599b0 0x7f358815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042727228 0 0) 0x7f3588159ec0 con 0x7f35880af0a0 2023-11-08T08:19:58.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35700035b0 con 0x7f35880af0a0 2023-11-08T08:19:58.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882413662 0 0) 0x7f358815aa60 con 0x7f35880b1650 2023-11-08T08:19:58.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3588159ec0 con 0x7f35880b1650 2023-11-08T08:19:58.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976605971 0 0) 0x7f35700035b0 con 0x7f35880af0a0 2023-11-08T08:19:58.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3585d9b700 1 -- 172.21.15.116:0/1394105721 >> v1:172.21.15.129:6789/0 conn(0x7f35880b1650 legacy=0x7f35880b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3585d9b700 1 -- 172.21.15.116:0/1394105721 >> v1:172.21.15.116:6789/0 conn(0x7f35880af0a0 legacy=0x7f358815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3585d9b700 1 -- 172.21.15.116:0/1394105721 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358815db80 con 0x7f35881599b0 2023-11-08T08:19:58.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f3585d9b700 1 --2- 172.21.15.116:0/1394105721 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35881599b0 0x7f358815d050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3578009870 tx=0x7f3578009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e3853e46f388afa server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.764+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800019f0 con 0x7f35881599b0 2023-11-08T08:19:58.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3578004030 con 0x7f35881599b0 2023-11-08T08:19:58.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358559a700 1 -- 172.21.15.116:0/1394105721 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35780043c0 con 0x7f35881599b0 2023-11-08T08:19:58.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 -- 172.21.15.116:0/1394105721 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35881599b0 msgr2=0x7f358815d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 --2- 172.21.15.116:0/1394105721 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35881599b0 0x7f358815d050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3578009870 tx=0x7f3578009b80 comp rx=0 tx=0).stop 2023-11-08T08:19:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 -- 172.21.15.116:0/1394105721 shutdown_connections 2023-11-08T08:19:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 --2- 172.21.15.116:0/1394105721 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35881599b0 0x7f358815d050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 -- 172.21.15.116:0/1394105721 >> 172.21.15.116:0/1394105721 conn(0x7f358806c010 msgr2=0x7f35880cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 -- 172.21.15.116:0/1394105721 shutdown_connections 2023-11-08T08:19:58.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.765+0000 7f358cb7a700 1 -- 172.21.15.116:0/1394105721 wait complete. 2023-11-08T08:19:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.766+0000 7f358cb7a700 1 Processor -- start 2023-11-08T08:19:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.766+0000 7f358cb7a700 1 -- start start 2023-11-08T08:19:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358cb7a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35880af0a0 0x7f358815c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358cb7a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 0x7f35881f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358cb7a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35881f2240 0x7f35881f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358cb7a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358815c8c0 con 0x7f35880b1650 2023-11-08T08:19:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358cb7a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35880c4080 con 0x7f35881f2240 2023-11-08T08:19:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358cb7a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35880c4360 con 0x7f35880af0a0 2023-11-08T08:19:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 0x7f35881f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 0x7f35881f15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35090/0 (socket says 172.21.15.116:35090) 2023-11-08T08:19:58.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 -- 172.21.15.116:0/4201577899 learned_addr learned my addr 172.21.15.116:0/4201577899 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:19:58.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358659c700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35880af0a0 0x7f358815c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3586d9d700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35881f2240 0x7f35881f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 -- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35880af0a0 msgr2=0x7f358815c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35880af0a0 0x7f358815c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 -- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35881f2240 msgr2=0x7f35881f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35881f2240 0x7f35881f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3585d9b700 1 -- 172.21.15.116:0/4201577899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35780094a0 con 0x7f35880b1650 2023-11-08T08:19:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f358659c700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35880af0a0 0x7f358815c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.767+0000 7f3586d9d700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35881f2240 0x7f35881f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:19:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.768+0000 7f3585d9b700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 0x7f35881f15c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f35780097c0 tx=0x7f3578004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.768+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578003db0 con 0x7f35880b1650 2023-11-08T08:19:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.768+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357801f070 con 0x7f35880b1650 2023-11-08T08:19:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.768+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358815d590 con 0x7f35880b1650 2023-11-08T08:19:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.768+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578011870 con 0x7f35880b1650 2023-11-08T08:19:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.768+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358815de60 con 0x7f35880b1650 2023-11-08T08:19:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.769+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35880743f0 con 0x7f35880b1650 2023-11-08T08:19:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.769+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f357802a020 con 0x7f35880b1650 2023-11-08T08:19:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.770+0000 7f3576ffd700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35680682f0 0x7f356806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:19:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.770+0000 7f358659c700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35680682f0 0x7f356806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:19:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.770+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3578087180 con 0x7f35880b1650 2023-11-08T08:19:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.770+0000 7f358659c700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35680682f0 0x7f356806a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f357c007fd0 tx=0x7f357c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:19:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.774+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35780539e0 con 0x7f35880b1650 2023-11-08T08:19:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.967+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f358806cd10 con 0x7f35880b1650 2023-11-08T08:19:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.968+0000 7f3576ffd700 1 -- 172.21.15.116:0/4201577899 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v32) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3578022380 con 0x7f35880b1650 2023-11-08T08:19:58.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35680682f0 msgr2=0x7f356806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35680682f0 0x7f356806a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f357c007fd0 tx=0x7f357c009040 comp rx=0 tx=0).stop 2023-11-08T08:19:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 msgr2=0x7f35881f15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:19:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 0x7f35881f15c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f35780097c0 tx=0x7f3578004750 comp rx=0 tx=0).stop 2023-11-08T08:19:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 shutdown_connections 2023-11-08T08:19:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f35680682f0 0x7f356806a7a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35881f2240 0x7f35881f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f35880af0a0 0x7f358815c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 --2- 172.21.15.116:0/4201577899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35880b1650 0x7f35881f15c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:19:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 >> 172.21.15.116:0/4201577899 conn(0x7f358806c010 msgr2=0x7f35880d0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:19:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 shutdown_connections 2023-11-08T08:19:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:19:58.970+0000 7f358cb7a700 1 -- 172.21.15.116:0/4201577899 wait complete. 2023-11-08T08:19:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:19:58.982 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-08T08:19:58.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:19:58.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-08T08:19:59.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[138147]: fuse finished with error 0 and tester_r 0 2023-11-08T08:19:59.020 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:20:05.122 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:20:05.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:05.123 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-08T08:20:05.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:05.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.315+0000 7f8082b9b700 1 Processor -- start 2023-11-08T08:20:05.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.315+0000 7f8082b9b700 1 -- start start 2023-11-08T08:20:05.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.316+0000 7f8082b9b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c0d0c60 0x7f807c0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.316+0000 7f8082b9b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f807c160e70 0x7f807c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.316+0000 7f8082b9b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c06e900 con 0x7f807c06e400 2023-11-08T08:20:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.316+0000 7f8082b9b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c160390 con 0x7f807c160e70 2023-11-08T08:20:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.316+0000 7f8082b9b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c160670 con 0x7f807c0d0c60 2023-11-08T08:20:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8081138700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f807c160e70 0x7f807c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8080937700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c0d0c60 0x7f807c0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f807bfff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f807c06e400 0x7f807c0d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:52322/0 (socket says 172.21.15.116:52322) 2023-11-08T08:20:05.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f807bfff700 1 -- 172.21.15.116:0/2973524923 learned_addr learned my addr 172.21.15.116:0/2973524923 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8081138700 1 -- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c0d0c60 msgr2=0x7f807c0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8081138700 1 --2- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c0d0c60 0x7f807c0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8081138700 1 -- 172.21.15.116:0/2973524923 >> v1:172.21.15.129:6789/0 conn(0x7f807c06e400 legacy=0x7f807c0d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8081138700 1 -- 172.21.15.116:0/2973524923 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c160950 con 0x7f807c160e70 2023-11-08T08:20:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f807b7fe700 1 -- 172.21.15.116:0/2973524923 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482701253 0 0) 0x7f807c06e900 con 0x7f807c06e400 2023-11-08T08:20:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.317+0000 7f8081138700 1 --2- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c160e70 0x7f807c15c970 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8074009d90 tx=0x7f807400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d77cbfba202f074a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f807b7fe700 1 -- 172.21.15.116:0/2973524923 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807400e070 con 0x7f807c160e70 2023-11-08T08:20:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f807b7fe700 1 -- 172.21.15.116:0/2973524923 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807400bb20 con 0x7f807c160e70 2023-11-08T08:20:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f807b7fe700 1 -- 172.21.15.116:0/2973524923 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074012430 con 0x7f807c160e70 2023-11-08T08:20:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 -- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c160e70 msgr2=0x7f807c15c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c160e70 0x7f807c15c970 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8074009d90 tx=0x7f807400b330 comp rx=0 tx=0).stop 2023-11-08T08:20:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 -- 172.21.15.116:0/2973524923 shutdown_connections 2023-11-08T08:20:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c160e70 0x7f807c15c970 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2973524923 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c0d0c60 0x7f807c0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 -- 172.21.15.116:0/2973524923 >> 172.21.15.116:0/2973524923 conn(0x7f807c06a270 msgr2=0x7f807c06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 -- 172.21.15.116:0/2973524923 shutdown_connections 2023-11-08T08:20:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.318+0000 7f8082b9b700 1 -- 172.21.15.116:0/2973524923 wait complete. 2023-11-08T08:20:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.320+0000 7f8082b9b700 1 Processor -- start 2023-11-08T08:20:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.320+0000 7f8082b9b700 1 -- start start 2023-11-08T08:20:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8082b9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c06e400 0x7f807c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8082b9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 0x7f807c15b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8082b9b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c160e70 0x7f807c15b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8082b9b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c1f6e30 con 0x7f807c0d0c60 2023-11-08T08:20:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8082b9b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c159660 con 0x7f807c06e400 2023-11-08T08:20:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8082b9b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c147540 con 0x7f807c160e70 2023-11-08T08:20:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 0x7f807c15b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 0x7f807c15b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56554/0 (socket says 172.21.15.116:56554) 2023-11-08T08:20:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 -- 172.21.15.116:0/2644755833 learned_addr learned my addr 172.21.15.116:0/2644755833 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8081138700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c160e70 0x7f807c15b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f8080937700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c06e400 0x7f807c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 -- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c160e70 msgr2=0x7f807c15b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c160e70 0x7f807c15b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 -- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c06e400 msgr2=0x7f807c159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c06e400 0x7f807c159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 -- 172.21.15.116:0/2644755833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074009a70 con 0x7f807c0d0c60 2023-11-08T08:20:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f807bfff700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 0x7f807c15b250 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f806800c540 tx=0x7f806800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068004c80 con 0x7f807c0d0c60 2023-11-08T08:20:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.321+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8068004de0 con 0x7f807c0d0c60 2023-11-08T08:20:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.322+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068007310 con 0x7f807c0d0c60 2023-11-08T08:20:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.322+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c15ceb0 con 0x7f807c0d0c60 2023-11-08T08:20:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.322+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c0d0740 con 0x7f807c0d0c60 2023-11-08T08:20:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.322+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c15d040 con 0x7f807c0d0c60 2023-11-08T08:20:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.323+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f806801a020 con 0x7f807c0d0c60 2023-11-08T08:20:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.323+0000 7f80797fa700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80600682f0 0x7f806006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.323+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f80680858d0 con 0x7f807c0d0c60 2023-11-08T08:20:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.323+0000 7f8080937700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80600682f0 0x7f806006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.324+0000 7f8080937700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80600682f0 0x7f806006a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8070009b70 tx=0x7f8070008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.328+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80680521e0 con 0x7f807c0d0c60 2023-11-08T08:20:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.551+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807c0743f0 con 0x7f807c0d0c60 2023-11-08T08:20:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.552+0000 7f80797fa700 1 -- 172.21.15.116:0/2644755833 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f8068055800 con 0x7f807c0d0c60 2023-11-08T08:20:05.553 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:05.554 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:20:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80600682f0 msgr2=0x7f806006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80600682f0 0x7f806006a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8070009b70 tx=0x7f8070008040 comp rx=0 tx=0).stop 2023-11-08T08:20:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 msgr2=0x7f807c15b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 0x7f807c15b250 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f806800c540 tx=0x7f806800c900 comp rx=0 tx=0).stop 2023-11-08T08:20:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 shutdown_connections 2023-11-08T08:20:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f807c160e70 0x7f807c15b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80600682f0 0x7f806006a7a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f807c06e400 0x7f807c159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 --2- 172.21.15.116:0/2644755833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c0d0c60 0x7f807c15b250 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 >> 172.21.15.116:0/2644755833 conn(0x7f807c06a270 msgr2=0x7f807c152550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 shutdown_connections 2023-11-08T08:20:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.554+0000 7f8082b9b700 1 -- 172.21.15.116:0/2644755833 wait complete. 2023-11-08T08:20:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:20:05.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2023-11-08T08:20:05.717 INFO:teuthology.orchestra.run.smithi116.stdout:{"message":"","return_code":0} 2023-11-08T08:20:05.717 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-08T08:20:05.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 Processor -- start 2023-11-08T08:20:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 -- start start 2023-11-08T08:20:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5dc0ce040 0x7ff5dc15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc0b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0b2110 con 0x7ff5dc142db0 2023-11-08T08:20:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc13fc90 con 0x7ff5dc0ce040 2023-11-08T08:20:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.837+0000 7ff5e1ac5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc13f480 con 0x7ff5dc0b1770 2023-11-08T08:20:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5db7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5dc0ce040 0x7ff5dc15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5d3fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff5dc142db0 0x7ff5dc13ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38366/0 (socket says 172.21.15.116:38366) 2023-11-08T08:20:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5d3fff700 1 -- 172.21.15.116:0/461478819 learned_addr learned my addr 172.21.15.116:0/461478819 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5db7fe700 1 -- 172.21.15.116:0/461478819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 msgr2=0x7ff5dc0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5db7fe700 1 --2- 172.21.15.116:0/461478819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc0b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5db7fe700 1 -- 172.21.15.116:0/461478819 >> v1:172.21.15.116:6789/0 conn(0x7ff5dc142db0 legacy=0x7ff5dc13ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5db7fe700 1 -- 172.21.15.116:0/461478819 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc13f6d0 con 0x7ff5dc0ce040 2023-11-08T08:20:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5daffd700 1 -- 172.21.15.116:0/461478819 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275955133 0 0) 0x7ff5dc0b2110 con 0x7ff5dc142db0 2023-11-08T08:20:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5db7fe700 1 --2- 172.21.15.116:0/461478819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc15d900 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009840 tx=0x7ff5cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6861dbc03fc14b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5daffd700 1 -- 172.21.15.116:0/461478819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011830 con 0x7ff5dc0ce040 2023-11-08T08:20:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5daffd700 1 -- 172.21.15.116:0/461478819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc011990 con 0x7ff5dc0ce040 2023-11-08T08:20:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.838+0000 7ff5daffd700 1 -- 172.21.15.116:0/461478819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011d20 con 0x7ff5dc0ce040 2023-11-08T08:20:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/461478819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc0ce040 msgr2=0x7ff5dc15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/461478819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc15d900 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009840 tx=0x7ff5cc009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/461478819 shutdown_connections 2023-11-08T08:20:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/461478819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc15d900 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/461478819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc0b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/461478819 >> 172.21.15.116:0/461478819 conn(0x7ff5dc06c010 msgr2=0x7ff5dc0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/461478819 shutdown_connections 2023-11-08T08:20:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.839+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/461478819 wait complete. 2023-11-08T08:20:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.840+0000 7ff5e1ac5700 1 Processor -- start 2023-11-08T08:20:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.840+0000 7ff5e1ac5700 1 -- start start 2023-11-08T08:20:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5e1ac5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc1f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5e1ac5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc1f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5e1ac5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 0x7ff5dc151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5e1ac5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc1f4940 con 0x7ff5dc142db0 2023-11-08T08:20:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5e1ac5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc1f0ac0 con 0x7ff5dc0ce040 2023-11-08T08:20:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5e1ac5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc152340 con 0x7ff5dc0b1770 2023-11-08T08:20:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5dbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 0x7ff5dc151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5dbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 0x7ff5dc151e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56566/0 (socket says 172.21.15.116:56566) 2023-11-08T08:20:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.841+0000 7ff5dbfff700 1 -- 172.21.15.116:0/3113555916 learned_addr learned my addr 172.21.15.116:0/3113555916 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5d3fff700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc1f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5db7fe700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc1f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5dbfff700 1 -- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 msgr2=0x7ff5dc1f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5dbfff700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc1f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5dbfff700 1 -- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc0ce040 msgr2=0x7ff5dc1f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5dbfff700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc1f4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5dbfff700 1 -- 172.21.15.116:0/3113555916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc0094a0 con 0x7ff5dc142db0 2023-11-08T08:20:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5db7fe700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc1f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5dbfff700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 0x7ff5dc151e00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400b970 tx=0x7ff5d400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400c9b0 con 0x7ff5dc142db0 2023-11-08T08:20:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d400cb30 con 0x7ff5dc142db0 2023-11-08T08:20:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc15d100 con 0x7ff5dc142db0 2023-11-08T08:20:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4014440 con 0x7ff5dc142db0 2023-11-08T08:20:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.842+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc13f9b0 con 0x7ff5dc142db0 2023-11-08T08:20:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.843+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc15d3e0 con 0x7ff5dc142db0 2023-11-08T08:20:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.843+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5d401b030 con 0x7ff5dc142db0 2023-11-08T08:20:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.843+0000 7ff5d8ff9700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.844+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff5d4085930 con 0x7ff5dc142db0 2023-11-08T08:20:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.844+0000 7ff5db7fe700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.844+0000 7ff5db7fe700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0b50b0 tx=0x7ff5cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:05.848+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5d4052190 con 0x7ff5dc142db0 2023-11-08T08:20:06.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.070+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5dc0743f0 con 0x7ff5dc142db0 2023-11-08T08:20:06.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.071+0000 7ff5d8ff9700 1 -- 172.21.15.116:0/3113555916 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7ff5d4019090 con 0x7ff5dc142db0 2023-11-08T08:20:06.073 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:06.073 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:20:06.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 msgr2=0x7ff5bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:06.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0b50b0 tx=0x7ff5cc008030 comp rx=0 tx=0).stop 2023-11-08T08:20:06.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 msgr2=0x7ff5dc151e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:06.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 0x7ff5dc151e00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400b970 tx=0x7ff5d400bd30 comp rx=0 tx=0).stop 2023-11-08T08:20:06.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 shutdown_connections 2023-11-08T08:20:06.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5dc142db0 0x7ff5dc151e00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:06.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5dc0b1770 0x7ff5dc1f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:06.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:06.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 --2- 172.21.15.116:0/3113555916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc0ce040 0x7ff5dc1f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:06.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 >> 172.21.15.116:0/3113555916 conn(0x7ff5dc06c010 msgr2=0x7ff5dc0caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:06.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 shutdown_connections 2023-11-08T08:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:06.073+0000 7ff5e1ac5700 1 -- 172.21.15.116:0/3113555916 wait complete. 2023-11-08T08:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:20:06.086 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json tag path /parentdir mytag 2023-11-08T08:20:06.227 INFO:teuthology.orchestra.run.smithi116.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2023-11-08T08:20:06.227 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2023-11-08T08:20:16.229 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2023-11-08T08:20:16.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.280+0000 7fced7b5f0c0 1 Processor -- start 2023-11-08T08:20:16.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.280+0000 7fced7b5f0c0 1 -- start start 2023-11-08T08:20:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fced7b5f0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c5f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fced7b5f0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5dc70 0x5568d1c602d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fced7b5f0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 0x5568d1c60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fced7b5f0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d1b3a000 con 0x5568d1c5e040 2023-11-08T08:20:16.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fced7b5f0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d1accfe0 con 0x5568d1bbce60 2023-11-08T08:20:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fced7b5f0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5568d1bdc800 con 0x5568d1c5dc70 2023-11-08T08:20:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fcece44c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 0x5568d1c60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fcece44c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 0x5568d1c60810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56066/0 (socket says 172.21.15.116:56066) 2023-11-08T08:20:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fcece44c700 1 -- 172.21.15.116:0/1176659830 learned_addr learned my addr 172.21.15.116:0/1176659830 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fcecdc4b700 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c5f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.281+0000 7fcecd44a700 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5dc70 0x5568d1c602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcece44c700 1 -- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5dc70 msgr2=0x5568d1c602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcece44c700 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5dc70 0x5568d1c602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcece44c700 1 -- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 msgr2=0x5568d1c5f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcece44c700 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c5f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcece44c700 1 -- 172.21.15.116:0/1176659830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4005d20 con 0x5568d1c5e040 2023-11-08T08:20:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcecdc4b700 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c5f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcece44c700 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 0x5568d1c60810 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcec8006e40 tx=0x7fcec8001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a99d75b89aecf3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcebffff700 1 -- 172.21.15.116:0/1176659830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8011440 con 0x5568d1c5e040 2023-11-08T08:20:16.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcebffff700 1 -- 172.21.15.116:0/1176659830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec80115a0 con 0x5568d1c5e040 2023-11-08T08:20:16.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fcebffff700 1 -- 172.21.15.116:0/1176659830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8011930 con 0x5568d1c5e040 2023-11-08T08:20:16.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1176659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 msgr2=0x5568d1c60810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.282+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 0x5568d1c60810 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcec8006e40 tx=0x7fcec8001930 comp rx=0 tx=0).stop 2023-11-08T08:20:16.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1176659830 shutdown_connections 2023-11-08T08:20:16.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5e040 0x5568d1c60810 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c5f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1176659830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5dc70 0x5568d1c602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1176659830 >> 172.21.15.116:0/1176659830 conn(0x5568d1bdd620 msgr2=0x5568d1bbe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1176659830 shutdown_connections 2023-11-08T08:20:16.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1176659830 wait complete. 2023-11-08T08:20:16.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 Processor -- start 2023-11-08T08:20:16.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.283+0000 7fced7b5f0c0 1 -- start start 2023-11-08T08:20:16.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fced7b5f0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c55bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fced7b5f0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5dc70 0x5568d1cf4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fced7b5f0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5e040 0x5568d1cf51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fced7b5f0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d1c57ed0 con 0x5568d1c5dc70 2023-11-08T08:20:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fced7b5f0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568d1c58730 con 0x5568d1bbce60 2023-11-08T08:20:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fced7b5f0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5568d1bdb730 con 0x5568d1c5e040 2023-11-08T08:20:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fcecd44a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5dc70 0x5568d1cf4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fcecd44a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5dc70 0x5568d1cf4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56074/0 (socket says 172.21.15.116:56074) 2023-11-08T08:20:16.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fcecd44a700 1 -- 172.21.15.116:0/1490970108 learned_addr learned my addr 172.21.15.116:0/1490970108 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fcecdc4b700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c55bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.284+0000 7fcece44c700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5e040 0x5568d1cf51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcecd44a700 1 -- 172.21.15.116:0/1490970108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5e040 msgr2=0x5568d1cf51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcecd44a700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5568d1c5e040 0x5568d1cf51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcecd44a700 1 -- 172.21.15.116:0/1490970108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 msgr2=0x5568d1c55bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcecd44a700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5568d1bbce60 0x5568d1c55bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcecd44a700 1 -- 172.21.15.116:0/1490970108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568d1ad84a0 con 0x5568d1c5dc70 2023-11-08T08:20:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcecd44a700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5dc70 0x5568d1cf4ca0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcec0001ed0 tx=0x7fcec000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcebeffd700 1 -- 172.21.15.116:0/1490970108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec000bd20 con 0x5568d1c5dc70 2023-11-08T08:20:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcebeffd700 1 -- 172.21.15.116:0/1490970108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0010410 con 0x5568d1c5dc70 2023-11-08T08:20:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5568d1ad8830 con 0x5568d1c5dc70 2023-11-08T08:20:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fcebeffd700 1 -- 172.21.15.116:0/1490970108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec000bea0 con 0x5568d1c5dc70 2023-11-08T08:20:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.285+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568d1ae5360 con 0x5568d1c5dc70 2023-11-08T08:20:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.286+0000 7fcebeffd700 1 -- 172.21.15.116:0/1490970108 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcec0020070 con 0x5568d1c5dc70 2023-11-08T08:20:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fcebeffd700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fceb0066c10 0x7fceb00690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fcebeffd700 1 -- 172.21.15.116:0/1490970108 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcec000a040 con 0x5568d1c5dc70 2023-11-08T08:20:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fcecdc4b700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fceb0066c10 0x7fceb00690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x5568d1c56770 0x5568d1c56b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 --> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5568d1c570b0 con 0x5568d1c56770 2023-11-08T08:20:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fcece44c700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x5568d1c56770 0x5568d1c56b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.287+0000 7fcecdc4b700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fceb0066c10 0x7fceb00690c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcec400a440 tx=0x7fcec400e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fcece44c700 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x5568d1c56770 0x5568d1c56b70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fcece44c700 1 -- 172.21.15.116:0/1490970108 <== osd.2 v2:172.21.15.116:6816/121156 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7fcec8018040 con 0x5568d1c56770 2023-11-08T08:20:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x5568d1c56770 msgr2=0x5568d1c56b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x5568d1c56770 0x5568d1c56b70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fceb0066c10 msgr2=0x7fceb00690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fceb0066c10 0x7fceb00690c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcec400a440 tx=0x7fcec400e000 comp rx=0 tx=0).stop 2023-11-08T08:20:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5dc70 msgr2=0x5568d1cf4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 --2- 172.21.15.116:0/1490970108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568d1c5dc70 0x5568d1cf4ca0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcec0001ed0 tx=0x7fcec000b4f0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fcece44c700 1 -- 172.21.15.116:0/1490970108 reap_dead start 2023-11-08T08:20:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 shutdown_connections 2023-11-08T08:20:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 >> 172.21.15.116:0/1490970108 conn(0x5568d1bdd620 msgr2=0x5568d1bbe760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 shutdown_connections 2023-11-08T08:20:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.288+0000 7fced7b5f0c0 1 -- 172.21.15.116:0/1490970108 wait complete. 2023-11-08T08:20:16.306 INFO:teuthology.orchestra.run.smithi116.stdout:mytag 2023-11-08T08:20:16.307 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2023-11-08T08:20:16.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 Processor -- start 2023-11-08T08:20:16.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 -- start start 2023-11-08T08:20:16.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c3602f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c36103a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c36103f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35fd8000 con 0x558c36030f80 2023-11-08T08:20:16.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35f6afe0 con 0x558c36030740 2023-11-08T08:20:16.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.355+0000 7fc3f00720c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c360fe580 con 0x558c36105b50 2023-11-08T08:20:16.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c36103a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c36103a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56090/0 (socket says 172.21.15.116:56090) 2023-11-08T08:20:16.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 -- 172.21.15.116:0/3223868646 learned_addr learned my addr 172.21.15.116:0/3223868646 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3e615e700 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c3602f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3e695f700 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c36103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 -- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 msgr2=0x558c36103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c36103f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 -- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 msgr2=0x558c3602f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c3602f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 -- 172.21.15.116:0/3223868646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8005d20 con 0x558c36030f80 2023-11-08T08:20:16.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3dffff700 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c36103a50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0001d50 tx=0x7fc3d000b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1b9d1cff629920 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3e515c700 1 -- 172.21.15.116:0/3223868646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000be20 con 0x558c36030f80 2023-11-08T08:20:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3e515c700 1 -- 172.21.15.116:0/3223868646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d000e470 con 0x558c36030f80 2023-11-08T08:20:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.356+0000 7fc3e515c700 1 -- 172.21.15.116:0/3223868646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000e820 con 0x558c36030f80 2023-11-08T08:20:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3223868646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 msgr2=0x558c36103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c36103a50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0001d50 tx=0x7fc3d000b5d0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3223868646 shutdown_connections 2023-11-08T08:20:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c36103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c3602f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3223868646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c36103a50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3223868646 >> 172.21.15.116:0/3223868646 conn(0x558c36078070 msgr2=0x558c36103100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3223868646 shutdown_connections 2023-11-08T08:20:16.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3223868646 wait complete. 2023-11-08T08:20:16.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 Processor -- start 2023-11-08T08:20:16.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.357+0000 7fc3f00720c0 1 -- start start 2023-11-08T08:20:16.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3f00720c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c361014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3f00720c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c360fb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3f00720c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c360fb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3f00720c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c360fbba0 con 0x558c36030f80 2023-11-08T08:20:16.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3f00720c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c360fd280 con 0x558c36030740 2023-11-08T08:20:16.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3f00720c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c360fd560 con 0x558c36105b50 2023-11-08T08:20:16.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3dffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c360fb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3dffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c360fb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56102/0 (socket says 172.21.15.116:56102) 2023-11-08T08:20:16.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3dffff700 1 -- 172.21.15.116:0/3668931914 learned_addr learned my addr 172.21.15.116:0/3668931914 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3e615e700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c361014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.358+0000 7fc3e695f700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c360fb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3dffff700 1 -- 172.21.15.116:0/3668931914 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 msgr2=0x558c360fb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3dffff700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558c36105b50 0x558c360fb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3dffff700 1 -- 172.21.15.116:0/3668931914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 msgr2=0x558c361014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3dffff700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c36030740 0x558c361014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3dffff700 1 -- 172.21.15.116:0/3668931914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4001e10 con 0x558c36030f80 2023-11-08T08:20:16.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3dffff700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c360fb030 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00011e0 tx=0x7fc3d0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3df7fe700 1 -- 172.21.15.116:0/3668931914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d001e070 con 0x558c36030f80 2023-11-08T08:20:16.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3df7fe700 1 -- 172.21.15.116:0/3668931914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0017420 con 0x558c36030f80 2023-11-08T08:20:16.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558c35f76830 con 0x558c36030f80 2023-11-08T08:20:16.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3df7fe700 1 -- 172.21.15.116:0/3668931914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000eea0 con 0x558c36030f80 2023-11-08T08:20:16.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.359+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c35f83360 con 0x558c36030f80 2023-11-08T08:20:16.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.360+0000 7fc3df7fe700 1 -- 172.21.15.116:0/3668931914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3d00177d0 con 0x558c36030f80 2023-11-08T08:20:16.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.360+0000 7fc3df7fe700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3c8066c10 0x7fc3c80690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3df7fe700 1 -- 172.21.15.116:0/3668931914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc3d0085ce0 con 0x558c36030f80 2023-11-08T08:20:16.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3e615e700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3c8066c10 0x7fc3c80690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x558c36101a00 0x558c360eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- osd_op(unknown.0.0:1 5.7 5:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558c360fead0 con 0x558c36101a00 2023-11-08T08:20:16.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3e695f700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x558c36101a00 0x558c360eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3e615e700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3c8066c10 0x7fc3c80690c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800a440 tx=0x7fc3d800f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3e695f700 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x558c36101a00 0x558c360eec70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.361+0000 7fc3e695f700 1 -- 172.21.15.116:0/3668931914 <== osd.3 v2:172.21.15.116:6808/121155 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7fc3e0006020 con 0x558c36101a00 2023-11-08T08:20:16.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x558c36101a00 msgr2=0x558c360eec70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x558c36101a00 0x558c360eec70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3c8066c10 msgr2=0x7fc3c80690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc3c8066c10 0x7fc3c80690c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800a440 tx=0x7fc3d800f000 comp rx=0 tx=0).stop 2023-11-08T08:20:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 msgr2=0x558c360fb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 --2- 172.21.15.116:0/3668931914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c36030f80 0x558c360fb030 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00011e0 tx=0x7fc3d0000ed0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3e695f700 1 -- 172.21.15.116:0/3668931914 reap_dead start 2023-11-08T08:20:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 shutdown_connections 2023-11-08T08:20:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 >> 172.21.15.116:0/3668931914 conn(0x558c36078070 msgr2=0x558c36104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 shutdown_connections 2023-11-08T08:20:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.362+0000 7fc3f00720c0 1 -- 172.21.15.116:0/3668931914 wait complete. 2023-11-08T08:20:16.383 INFO:teuthology.orchestra.run.smithi116.stdout:mytag 2023-11-08T08:20:16.383 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2023-11-08T08:20:16.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.428+0000 7ffbb10330c0 1 Processor -- start 2023-11-08T08:20:16.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.428+0000 7ffbb10330c0 1 -- start start 2023-11-08T08:20:16.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffbb10330c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d333d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffbb10330c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28da80 0x558b4d28e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffbb10330c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 0x558b4d28e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffbb10330c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b4d20a000 con 0x558b4d28def0 2023-11-08T08:20:16.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffbb10330c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b4d19cfe0 con 0x558b4d31b2c0 2023-11-08T08:20:16.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffbb10330c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b4d33db70 con 0x558b4d28da80 2023-11-08T08:20:16.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 0x558b4d28e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 0x558b4d28e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56120/0 (socket says 172.21.15.116:56120) 2023-11-08T08:20:16.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 -- 172.21.15.116:0/3556068593 learned_addr learned my addr 172.21.15.116:0/3556068593 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba691e700 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28da80 0x558b4d28e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba711f700 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d333d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 -- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28da80 msgr2=0x558b4d28e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28da80 0x558b4d28e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 -- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 msgr2=0x558b4d333d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d333d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 -- 172.21.15.116:0/3556068593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b4d1b5360 con 0x558b4d28def0 2023-11-08T08:20:16.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.429+0000 7ffba7920700 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 0x558b4d28e960 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffba0002230 tx=0x7ffba0008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440a798887394759 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffba591c700 1 -- 172.21.15.116:0/3556068593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011420 con 0x558b4d28def0 2023-11-08T08:20:16.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffba591c700 1 -- 172.21.15.116:0/3556068593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba001f070 con 0x558b4d28def0 2023-11-08T08:20:16.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffba591c700 1 -- 172.21.15.116:0/3556068593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011910 con 0x558b4d28def0 2023-11-08T08:20:16.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 -- 172.21.15.116:0/3556068593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 msgr2=0x558b4d28e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 0x558b4d28e960 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffba0002230 tx=0x7ffba0008000 comp rx=0 tx=0).stop 2023-11-08T08:20:16.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 -- 172.21.15.116:0/3556068593 shutdown_connections 2023-11-08T08:20:16.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28def0 0x558b4d28e960 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d333d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/3556068593 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28da80 0x558b4d28e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 -- 172.21.15.116:0/3556068593 >> 172.21.15.116:0/3556068593 conn(0x558b4d2ad620 msgr2=0x558b4d264d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 -- 172.21.15.116:0/3556068593 shutdown_connections 2023-11-08T08:20:16.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.430+0000 7ffbb10330c0 1 -- 172.21.15.116:0/3556068593 wait complete. 2023-11-08T08:20:16.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 Processor -- start 2023-11-08T08:20:16.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 -- start start 2023-11-08T08:20:16.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28da80 0x558b4d31a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28def0 0x558b4d337100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d337640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b4d31aab0 con 0x558b4d28da80 2023-11-08T08:20:16.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b4d33c060 con 0x558b4d31b2c0 2023-11-08T08:20:16.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffbb10330c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b4d33c900 con 0x558b4d28def0 2023-11-08T08:20:16.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffba711f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28da80 0x558b4d31a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffba711f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28da80 0x558b4d31a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56124/0 (socket says 172.21.15.116:56124) 2023-11-08T08:20:16.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.431+0000 7ffba711f700 1 -- 172.21.15.116:0/4019484506 learned_addr learned my addr 172.21.15.116:0/4019484506 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba691e700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28def0 0x558b4d337100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba7920700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d337640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba711f700 1 -- 172.21.15.116:0/4019484506 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28def0 msgr2=0x558b4d337100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba711f700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x558b4d28def0 0x558b4d337100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba711f700 1 -- 172.21.15.116:0/4019484506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 msgr2=0x558b4d337640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba711f700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558b4d31b2c0 0x558b4d337640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba711f700 1 -- 172.21.15.116:0/4019484506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb90001e10 con 0x558b4d28da80 2023-11-08T08:20:16.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba711f700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28da80 0x558b4d31a080 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ffb98002e30 tx=0x7ffb9800e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba491a700 1 -- 172.21.15.116:0/4019484506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9800ed20 con 0x558b4d28da80 2023-11-08T08:20:16.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba491a700 1 -- 172.21.15.116:0/4019484506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb98008040 con 0x558b4d28da80 2023-11-08T08:20:16.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558b4d33b130 con 0x558b4d28da80 2023-11-08T08:20:16.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffba491a700 1 -- 172.21.15.116:0/4019484506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98012650 con 0x558b4d28da80 2023-11-08T08:20:16.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.432+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b4d1a8830 con 0x558b4d28da80 2023-11-08T08:20:16.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.433+0000 7ffba491a700 1 -- 172.21.15.116:0/4019484506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb980127b0 con 0x558b4d28da80 2023-11-08T08:20:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffba491a700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffb94066c10 0x7ffb940690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffba491a700 1 -- 172.21.15.116:0/4019484506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ffb98084c70 con 0x558b4d28da80 2023-11-08T08:20:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffba691e700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffb94066c10 0x7ffb940690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x558b4d33cd90 0x558b4d335880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 --> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] -- osd_op(unknown.0.0:1 6.36 6:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558b4d3367d0 con 0x558b4d33cd90 2023-11-08T08:20:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffba7920700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x558b4d33cd90 0x558b4d335880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffba691e700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffb94066c10 0x7ffb940690c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c001820 tx=0x7ffb9c001a70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.434+0000 7ffba7920700 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x558b4d33cd90 0x558b4d335880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffba7920700 1 -- 172.21.15.116:0/4019484506 <== osd.1 v2:172.21.15.116:6800/121158 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7ffba002d040 con 0x558b4d33cd90 2023-11-08T08:20:16.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x558b4d33cd90 msgr2=0x558b4d335880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x558b4d33cd90 0x558b4d335880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffb94066c10 msgr2=0x7ffb940690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ffb94066c10 0x7ffb940690c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c001820 tx=0x7ffb9c001a70 comp rx=0 tx=0).stop 2023-11-08T08:20:16.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28da80 msgr2=0x558b4d31a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 --2- 172.21.15.116:0/4019484506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b4d28da80 0x558b4d31a080 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ffb98002e30 tx=0x7ffb9800e5b0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffba7920700 1 -- 172.21.15.116:0/4019484506 reap_dead start 2023-11-08T08:20:16.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 shutdown_connections 2023-11-08T08:20:16.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 >> 172.21.15.116:0/4019484506 conn(0x558b4d2ad620 msgr2=0x558b4d32be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 shutdown_connections 2023-11-08T08:20:16.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.435+0000 7ffbb10330c0 1 -- 172.21.15.116:0/4019484506 wait complete. 2023-11-08T08:20:16.454 INFO:teuthology.orchestra.run.smithi116.stdout:mytag 2023-11-08T08:20:16.454 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2023-11-08T08:20:16.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.498+0000 7f6125b1a0c0 1 Processor -- start 2023-11-08T08:20:16.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.498+0000 7f6125b1a0c0 1 -- start start 2023-11-08T08:20:16.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f6125b1a0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f6125b1a0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f308d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f6125b1a0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f308d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f6125b1a0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5f3005000 con 0x7f61040030b0 2023-11-08T08:20:16.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f6125b1a0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5f2f97fe0 con 0x55e5f312c0d0 2023-11-08T08:20:16.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f6125b1a0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5f305d240 con 0x55e5f312bc60 2023-11-08T08:20:16.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56138/0 (socket says 172.21.15.116:56138) 2023-11-08T08:20:16.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 -- 172.21.15.116:0/260529864 learned_addr learned my addr 172.21.15.116:0/260529864 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611b405700 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f308d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611c407700 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f308d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 -- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 msgr2=0x55e5f308d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f308d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 -- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 msgr2=0x55e5f308d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f308d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 -- 172.21.15.116:0/260529864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5f2fb0360 con 0x7f61040030b0 2023-11-08T08:20:16.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611c407700 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f308d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:16.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.499+0000 7f611bc06700 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312d550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f610c000fb0 tx=0x7f610c00d650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d522575e306792 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f611a403700 1 -- 172.21.15.116:0/260529864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c00dd80 con 0x7f61040030b0 2023-11-08T08:20:16.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f611a403700 1 -- 172.21.15.116:0/260529864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c00f450 con 0x7f61040030b0 2023-11-08T08:20:16.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f611a403700 1 -- 172.21.15.116:0/260529864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c00f7e0 con 0x7f61040030b0 2023-11-08T08:20:16.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/260529864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 msgr2=0x55e5f312d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312d550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f610c000fb0 tx=0x7f610c00d650 comp rx=0 tx=0).stop 2023-11-08T08:20:16.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/260529864 shutdown_connections 2023-11-08T08:20:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f308d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312d550 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/260529864 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f308d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/260529864 >> 172.21.15.116:0/260529864 conn(0x55e5f30a5070 msgr2=0x55e5f305df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/260529864 shutdown_connections 2023-11-08T08:20:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.500+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/260529864 wait complete. 2023-11-08T08:20:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 Processor -- start 2023-11-08T08:20:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 -- start start 2023-11-08T08:20:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f31c1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f3125390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5f31bfaf0 con 0x7f61040030b0 2023-11-08T08:20:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5f31c0a80 con 0x55e5f312c0d0 2023-11-08T08:20:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.501+0000 7f6125b1a0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5f31c0d60 con 0x55e5f312bc60 2023-11-08T08:20:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56142/0 (socket says 172.21.15.116:56142) 2023-11-08T08:20:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 -- 172.21.15.116:0/2733950625 learned_addr learned my addr 172.21.15.116:0/2733950625 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611b405700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f3125390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611bc06700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f31c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 -- 172.21.15.116:0/2733950625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 msgr2=0x55e5f31c1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e5f312bc60 0x55e5f31c1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 -- 172.21.15.116:0/2733950625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 msgr2=0x55e5f3125390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5f312c0d0 0x55e5f3125390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 -- 172.21.15.116:0/2733950625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5f2fb0120 con 0x7f61040030b0 2023-11-08T08:20:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f611c407700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312f870 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x55e5f30a83e0 tx=0x7f6114011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f6119401700 1 -- 172.21.15.116:0/2733950625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114011ee0 con 0x7f61040030b0 2023-11-08T08:20:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f6119401700 1 -- 172.21.15.116:0/2733950625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6114014660 con 0x7f61040030b0 2023-11-08T08:20:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e5f3138d00 con 0x7f61040030b0 2023-11-08T08:20:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f6119401700 1 -- 172.21.15.116:0/2733950625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114014830 con 0x7f61040030b0 2023-11-08T08:20:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.502+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5f2fa3830 con 0x7f61040030b0 2023-11-08T08:20:16.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.503+0000 7f6119401700 1 -- 172.21.15.116:0/2733950625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6114014990 con 0x7f61040030b0 2023-11-08T08:20:16.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f6119401700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6108066c10 0x7f61080690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f6119401700 1 -- 172.21.15.116:0/2733950625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6114084ac0 con 0x7f61040030b0 2023-11-08T08:20:16.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f611bc06700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6108066c10 0x7f61080690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x55e5f312b430 0x55e5f30a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 --> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] -- osd_op(unknown.0.0:1 6.19 6:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e5f3132b40 con 0x55e5f312b430 2023-11-08T08:20:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f611b405700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x55e5f312b430 0x55e5f30a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f611bc06700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6108066c10 0x7f61080690c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f610c012040 tx=0x7f610c008000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.504+0000 7f611b405700 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x55e5f312b430 0x55e5f30a2990 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f611b405700 1 -- 172.21.15.116:0/2733950625 <== osd.0 v2:172.21.15.116:6824/121157 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f6110016070 con 0x55e5f312b430 2023-11-08T08:20:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x55e5f312b430 msgr2=0x55e5f30a2990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x55e5f312b430 0x55e5f30a2990 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6108066c10 msgr2=0x7f61080690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6108066c10 0x7f61080690c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f610c012040 tx=0x7f610c008000 comp rx=0 tx=0).stop 2023-11-08T08:20:16.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 msgr2=0x55e5f312f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 --2- 172.21.15.116:0/2733950625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61040030b0 0x55e5f312f870 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x55e5f30a83e0 tx=0x7f6114011850 comp rx=0 tx=0).stop 2023-11-08T08:20:16.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f611c407700 1 -- 172.21.15.116:0/2733950625 reap_dead start 2023-11-08T08:20:16.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 shutdown_connections 2023-11-08T08:20:16.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 >> 172.21.15.116:0/2733950625 conn(0x55e5f30a5070 msgr2=0x55e5f3089760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 shutdown_connections 2023-11-08T08:20:16.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.505+0000 7f6125b1a0c0 1 -- 172.21.15.116:0/2733950625 wait complete. 2023-11-08T08:20:16.538 INFO:teuthology.orchestra.run.smithi116.stdout:mytag 2023-11-08T08:20:16.538 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2023-11-08T08:20:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.582+0000 7f7c9190c0c0 1 Processor -- start 2023-11-08T08:20:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 -- start start 2023-11-08T08:20:16.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7c70002880 0x564397dd5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564397d2bba0 0x564397cfa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 0x564397dbabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564397ca3000 con 0x564397cfa8b0 2023-11-08T08:20:16.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x564397c35fe0 con 0x564397d2bba0 2023-11-08T08:20:16.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c9190c0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x564397dd5c90 con 0x7f7c70002880 2023-11-08T08:20:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c881f9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 0x564397dbabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c881f9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 0x564397dbabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56166/0 (socket says 172.21.15.116:56166) 2023-11-08T08:20:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c881f9700 1 -- 172.21.15.116:0/3963548883 learned_addr learned my addr 172.21.15.116:0/3963548883 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c871f7700 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564397d2bba0 0x564397cfa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.583+0000 7f7c879f8700 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7c70002880 0x564397dd5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c881f9700 1 -- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7c70002880 msgr2=0x564397dd5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c881f9700 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7c70002880 0x564397dd5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c881f9700 1 -- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564397d2bba0 msgr2=0x564397cfa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c881f9700 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564397d2bba0 0x564397cfa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c881f9700 1 -- 172.21.15.116:0/3963548883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564397c4e360 con 0x564397cfa8b0 2023-11-08T08:20:16.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c871f7700 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564397d2bba0 0x564397cfa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:16.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c881f9700 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 0x564397dbabe0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x564397ce8be0 tx=0x7f7c80006340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ebfb0457ab9e01 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c861f5700 1 -- 172.21.15.116:0/3963548883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80001430 con 0x564397cfa8b0 2023-11-08T08:20:16.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c861f5700 1 -- 172.21.15.116:0/3963548883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80006de0 con 0x564397cfa8b0 2023-11-08T08:20:16.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c861f5700 1 -- 172.21.15.116:0/3963548883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80003690 con 0x564397cfa8b0 2023-11-08T08:20:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/3963548883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 msgr2=0x564397dbabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 0x564397dbabe0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x564397ce8be0 tx=0x7f7c80006340 comp rx=0 tx=0).stop 2023-11-08T08:20:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/3963548883 shutdown_connections 2023-11-08T08:20:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397cfa8b0 0x564397dbabe0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7c70002880 0x564397dd5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/3963548883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564397d2bba0 0x564397cfa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/3963548883 >> 172.21.15.116:0/3963548883 conn(0x564397d45da0 msgr2=0x564397dd65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/3963548883 shutdown_connections 2023-11-08T08:20:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.584+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/3963548883 wait complete. 2023-11-08T08:20:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.585+0000 7f7c9190c0c0 1 Processor -- start 2023-11-08T08:20:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.585+0000 7f7c9190c0c0 1 -- start start 2023-11-08T08:20:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564397cfa8b0 0x564397dd2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397d2bba0 0x564397d24220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c70002880 0x564397d2a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564397d24b40 con 0x564397d2bba0 2023-11-08T08:20:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x564397db9df0 con 0x7f7c70002880 2023-11-08T08:20:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x564397dba000 con 0x564397cfa8b0 2023-11-08T08:20:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397d2bba0 0x564397d24220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397d2bba0 0x564397d24220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56182/0 (socket says 172.21.15.116:56182) 2023-11-08T08:20:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 -- 172.21.15.116:0/1091271938 learned_addr learned my addr 172.21.15.116:0/1091271938 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c879f8700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564397cfa8b0 0x564397dd2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c881f9700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c70002880 0x564397d2a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 -- 172.21.15.116:0/1091271938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564397cfa8b0 msgr2=0x564397dd2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564397cfa8b0 0x564397dd2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 -- 172.21.15.116:0/1091271938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c70002880 msgr2=0x564397d2a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c70002880 0x564397d2a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 -- 172.21.15.116:0/1091271938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564397c4e120 con 0x564397d2bba0 2023-11-08T08:20:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c871f7700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397d2bba0 0x564397d24220 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000a520 tx=0x7f7c78003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c851f3700 1 -- 172.21.15.116:0/1091271938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78008ac0 con 0x564397d2bba0 2023-11-08T08:20:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564397cf9f00 con 0x564397d2bba0 2023-11-08T08:20:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c851f3700 1 -- 172.21.15.116:0/1091271938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c78008c20 con 0x564397d2bba0 2023-11-08T08:20:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c851f3700 1 -- 172.21.15.116:0/1091271938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78015420 con 0x564397d2bba0 2023-11-08T08:20:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.586+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564397c41830 con 0x564397d2bba0 2023-11-08T08:20:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c851f3700 1 -- 172.21.15.116:0/1091271938 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c78027030 con 0x564397d2bba0 2023-11-08T08:20:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:20:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c851f3700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7c74066c10 0x7f7c740690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c851f3700 1 -- 172.21.15.116:0/1091271938 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7c78085420 con 0x564397d2bba0 2023-11-08T08:20:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c879f8700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7c74066c10 0x7f7c740690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x564397d2a8c0 0x564397e6deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 --> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] -- osd_op(unknown.0.0:1 6.12 6:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564397d40550 con 0x564397d2a8c0 2023-11-08T08:20:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c881f9700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x564397d2a8c0 0x564397e6deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.588+0000 7f7c879f8700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7c74066c10 0x7f7c740690c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0034b0 tx=0x7f7c7c0032f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c881f9700 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x564397d2a8c0 0x564397e6deb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c881f9700 1 -- 172.21.15.116:0/1091271938 <== osd.0 v2:172.21.15.116:6824/121157 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f7c80018020 con 0x564397d2a8c0 2023-11-08T08:20:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2023-11-08T08:20:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x564397d2a8c0 msgr2=0x564397e6deb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x564397d2a8c0 0x564397e6deb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7c74066c10 msgr2=0x7f7c740690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7c74066c10 0x7f7c740690c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0034b0 tx=0x7f7c7c0032f0 comp rx=0 tx=0).stop 2023-11-08T08:20:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397d2bba0 msgr2=0x564397d24220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 --2- 172.21.15.116:0/1091271938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564397d2bba0 0x564397d24220 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000a520 tx=0x7f7c78003020 comp rx=0 tx=0).stop 2023-11-08T08:20:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c881f9700 1 -- 172.21.15.116:0/1091271938 reap_dead start 2023-11-08T08:20:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 shutdown_connections 2023-11-08T08:20:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 >> 172.21.15.116:0/1091271938 conn(0x564397d45da0 msgr2=0x564397cfac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 shutdown_connections 2023-11-08T08:20:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:16.589+0000 7f7c9190c0c0 1 -- 172.21.15.116:0/1091271938 wait complete. 2023-11-08T08:20:16.608 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-08T08:20:16.608 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:20:16.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:20:16.697 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-08T08:20:16.697 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:20:16.698 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:16.716 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:20:16.717 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:20:16.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:16.718 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-08T08:20:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:20:16.775 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-08T08:20:16.776 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:20:16.776 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:16.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:20:16.798 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:20:16.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:16.799 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-08T08:20:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:20:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-11-08T08:20:16.858 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:20:16.858 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:16.876 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:20:16.876 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:20:16.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:16.877 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-08T08:20:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:20:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-11-08T08:20:16.935 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:20:16.936 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:16.956 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:20:16.957 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:20:16.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:16.958 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-08T08:20:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:20:17.017 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-11-08T08:20:17.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.139+0000 7f5d51589700 1 Processor -- start 2023-11-08T08:20:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.139+0000 7f5d51589700 1 -- start start 2023-11-08T08:20:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d51589700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d4c070d90 0x7f5d4c0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d51589700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c151a50 con 0x7f5d4c06a270 2023-11-08T08:20:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d51589700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c15b440 con 0x7f5d4c15e0b0 2023-11-08T08:20:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d51589700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c071ac0 con 0x7f5d4c070d90 2023-11-08T08:20:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d4a7fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d4c070d90 0x7f5d4c0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d4affd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5d4c06a270 0x7f5d4c06a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58376/0 (socket says 172.21.15.116:58376) 2023-11-08T08:20:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d4affd700 1 -- 172.21.15.116:0/2668862452 learned_addr learned my addr 172.21.15.116:0/2668862452 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d4a7fc700 1 -- 172.21.15.116:0/2668862452 >> v1:172.21.15.129:6789/0 conn(0x7f5d4c15e0b0 legacy=0x7f5d4c15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d4a7fc700 1 -- 172.21.15.116:0/2668862452 >> v1:172.21.15.116:6789/0 conn(0x7f5d4c06a270 legacy=0x7f5d4c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d49ffb700 1 -- 172.21.15.116:0/2668862452 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150288079 0 0) 0x7f5d4c151a50 con 0x7f5d4c06a270 2023-11-08T08:20:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.140+0000 7f5d4a7fc700 1 -- 172.21.15.116:0/2668862452 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c15adf0 con 0x7f5d4c070d90 2023-11-08T08:20:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d4a7fc700 1 --2- 172.21.15.116:0/2668862452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c070d90 0x7f5d4c0711e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009840 tx=0x7f5d3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae8893f8573ea94 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d49ffb700 1 -- 172.21.15.116:0/2668862452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c011830 con 0x7f5d4c070d90 2023-11-08T08:20:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d49ffb700 1 -- 172.21.15.116:0/2668862452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c011990 con 0x7f5d4c070d90 2023-11-08T08:20:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d49ffb700 1 -- 172.21.15.116:0/2668862452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c011d20 con 0x7f5d4c070d90 2023-11-08T08:20:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 -- 172.21.15.116:0/2668862452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c070d90 msgr2=0x7f5d4c0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 --2- 172.21.15.116:0/2668862452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c070d90 0x7f5d4c0711e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009840 tx=0x7f5d3c009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 -- 172.21.15.116:0/2668862452 shutdown_connections 2023-11-08T08:20:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 --2- 172.21.15.116:0/2668862452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c070d90 0x7f5d4c0711e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 -- 172.21.15.116:0/2668862452 >> 172.21.15.116:0/2668862452 conn(0x7f5d4c06c010 msgr2=0x7f5d4c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:17.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 -- 172.21.15.116:0/2668862452 shutdown_connections 2023-11-08T08:20:17.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.141+0000 7f5d51589700 1 -- 172.21.15.116:0/2668862452 wait complete. 2023-11-08T08:20:17.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 Processor -- start 2023-11-08T08:20:17.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 -- start start 2023-11-08T08:20:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 0x7f5d4c14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d4c070d90 0x7f5d4c150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c15e0b0 0x7f5d4c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0b4d60 con 0x7f5d4c06a270 2023-11-08T08:20:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0b5040 con 0x7f5d4c15e0b0 2023-11-08T08:20:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d51589700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0b5320 con 0x7f5d4c070d90 2023-11-08T08:20:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d4affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 0x7f5d4c14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d4affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 0x7f5d4c14c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56202/0 (socket says 172.21.15.116:56202) 2023-11-08T08:20:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.143+0000 7f5d4affd700 1 -- 172.21.15.116:0/3540017598 learned_addr learned my addr 172.21.15.116:0/3540017598 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4b7fe700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c15e0b0 0x7f5d4c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4a7fc700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d4c070d90 0x7f5d4c150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4affd700 1 -- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d4c070d90 msgr2=0x7f5d4c150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4affd700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d4c070d90 0x7f5d4c150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4affd700 1 -- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c15e0b0 msgr2=0x7f5d4c150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4affd700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c15e0b0 0x7f5d4c150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4affd700 1 -- 172.21.15.116:0/3540017598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0094a0 con 0x7f5d4c06a270 2023-11-08T08:20:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4b7fe700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c15e0b0 0x7f5d4c150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d4affd700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 0x7f5d4c14c1c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000c520 tx=0x7f5d4000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40004c80 con 0x7f5d4c06a270 2023-11-08T08:20:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d40004e00 con 0x7f5d4c06a270 2023-11-08T08:20:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c15aac0 con 0x7f5d4c06a270 2023-11-08T08:20:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40007310 con 0x7f5d4c06a270 2023-11-08T08:20:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.144+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c15b0d0 con 0x7f5d4c06a270 2023-11-08T08:20:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.145+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c071720 con 0x7f5d4c06a270 2023-11-08T08:20:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.145+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d4001a020 con 0x7f5d4c06a270 2023-11-08T08:20:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.146+0000 7f5d337fe700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5d380682f0 0x7f5d3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.146+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5d40085910 con 0x7f5d4c06a270 2023-11-08T08:20:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.146+0000 7f5d4a7fc700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5d380682f0 0x7f5d3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.146+0000 7f5d4a7fc700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5d380682f0 0x7f5d3806a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c15a460 tx=0x7f5d3c01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.150+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d40052170 con 0x7f5d4c06a270 2023-11-08T08:20:17.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.370+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d4c0743f0 con 0x7f5d4c06a270 2023-11-08T08:20:17.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.371+0000 7f5d337fe700 1 -- 172.21.15.116:0/3540017598 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f5d40055790 con 0x7f5d4c06a270 2023-11-08T08:20:17.372 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:17.373 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:20:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5d380682f0 msgr2=0x7f5d3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5d380682f0 0x7f5d3806a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c15a460 tx=0x7f5d3c01c040 comp rx=0 tx=0).stop 2023-11-08T08:20:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 msgr2=0x7f5d4c14c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 0x7f5d4c14c1c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000c520 tx=0x7f5d4000c8e0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 shutdown_connections 2023-11-08T08:20:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5d380682f0 0x7f5d3806a7a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c15e0b0 0x7f5d4c150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c06a270 0x7f5d4c14c1c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 --2- 172.21.15.116:0/3540017598 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5d4c070d90 0x7f5d4c150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 >> 172.21.15.116:0/3540017598 conn(0x7f5d4c06c010 msgr2=0x7f5d4c15d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 shutdown_connections 2023-11-08T08:20:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.373+0000 7f5d51589700 1 -- 172.21.15.116:0/3540017598 wait complete. 2023-11-08T08:20:17.377 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:20:17.386 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:17.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.505+0000 7ff5caf62700 1 Processor -- start 2023-11-08T08:20:17.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5caf62700 1 -- start start 2023-11-08T08:20:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5caf62700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5c406a270 0x7ff5c406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5caf62700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff5c406e570 0x7ff5c4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5caf62700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c406ab80 con 0x7ff5c415cc80 2023-11-08T08:20:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5caf62700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c40b2490 con 0x7ff5c406a270 2023-11-08T08:20:17.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5caf62700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c406e990 con 0x7ff5c406e570 2023-11-08T08:20:17.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5c8cfe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5c406a270 0x7ff5c406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5c8cfe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5c406a270 0x7ff5c406a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56208/0 (socket says 172.21.15.116:56208) 2023-11-08T08:20:17.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.506+0000 7ff5c8cfe700 1 -- 172.21.15.116:0/2513573570 learned_addr learned my addr 172.21.15.116:0/2513573570 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:17.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c94ff700 1 --2- 172.21.15.116:0/2513573570 >> v2:172.21.15.129:3300/0 conn(0x7ff5c406e570 0x7ff5c4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c8cfe700 1 -- 172.21.15.116:0/2513573570 >> v2:172.21.15.129:3300/0 conn(0x7ff5c406e570 msgr2=0x7ff5c4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:17.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c8cfe700 1 --2- 172.21.15.116:0/2513573570 >> v2:172.21.15.129:3300/0 conn(0x7ff5c406e570 0x7ff5c4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c8cfe700 1 -- 172.21.15.116:0/2513573570 >> v1:172.21.15.129:6789/0 conn(0x7ff5c415cc80 legacy=0x7ff5c406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:17.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c8cfe700 1 -- 172.21.15.116:0/2513573570 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c40b1810 con 0x7ff5c406a270 2023-11-08T08:20:17.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c8cfe700 1 --2- 172.21.15.116:0/2513573570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c406a270 0x7ff5c406a640 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff5c406ee30 tx=0x7ff5b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8084b0708ca6e164 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c37fe700 1 -- 172.21.15.116:0/2513573570 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8004230 con 0x7ff5c406a270 2023-11-08T08:20:17.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c37fe700 1 -- 172.21.15.116:0/2513573570 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b8004390 con 0x7ff5c406a270 2023-11-08T08:20:17.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.507+0000 7ff5c37fe700 1 -- 172.21.15.116:0/2513573570 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b80036a0 con 0x7ff5c406a270 2023-11-08T08:20:17.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 -- 172.21.15.116:0/2513573570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c406a270 msgr2=0x7ff5c406a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 --2- 172.21.15.116:0/2513573570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c406a270 0x7ff5c406a640 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff5c406ee30 tx=0x7ff5b8009c00 comp rx=0 tx=0).stop 2023-11-08T08:20:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 -- 172.21.15.116:0/2513573570 shutdown_connections 2023-11-08T08:20:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 --2- 172.21.15.116:0/2513573570 >> v2:172.21.15.129:3300/0 conn(0x7ff5c406e570 0x7ff5c4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 --2- 172.21.15.116:0/2513573570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c406a270 0x7ff5c406a640 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 -- 172.21.15.116:0/2513573570 >> 172.21.15.116:0/2513573570 conn(0x7ff5c406c010 msgr2=0x7ff5c40cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 -- 172.21.15.116:0/2513573570 shutdown_connections 2023-11-08T08:20:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.508+0000 7ff5caf62700 1 -- 172.21.15.116:0/2513573570 wait complete. 2023-11-08T08:20:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.509+0000 7ff5caf62700 1 Processor -- start 2023-11-08T08:20:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.509+0000 7ff5caf62700 1 -- start start 2023-11-08T08:20:17.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c406a270 0x7ff5c414b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5c406e570 0x7ff5c41e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 0x7ff5c40b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40b52c0 con 0x7ff5c415cc80 2023-11-08T08:20:17.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c414b8a0 con 0x7ff5c406e570 2023-11-08T08:20:17.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c41f6850 con 0x7ff5c406a270 2023-11-08T08:20:17.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 0x7ff5c40b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 0x7ff5c40b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56210/0 (socket says 172.21.15.116:56210) 2023-11-08T08:20:17.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 -- 172.21.15.116:0/4196084138 learned_addr learned my addr 172.21.15.116:0/4196084138 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:17.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c8cfe700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c406a270 0x7ff5c414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 -- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c406a270 msgr2=0x7ff5c414b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c3fff700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5c406e570 0x7ff5c41e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c406a270 0x7ff5c414b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 -- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5c406e570 msgr2=0x7ff5c41e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5c406e570 0x7ff5c41e6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 -- 172.21.15.116:0/4196084138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8009550 con 0x7ff5c415cc80 2023-11-08T08:20:17.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c8cfe700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c406a270 0x7ff5c414b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:17.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c94ff700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 0x7ff5c40b4700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc007e90 tx=0x7ff5bc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc00ed80 con 0x7ff5c415cc80 2023-11-08T08:20:17.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc014420 con 0x7ff5c415cc80 2023-11-08T08:20:17.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0147d0 con 0x7ff5c415cc80 2023-11-08T08:20:17.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c40b1dd0 con 0x7ff5c415cc80 2023-11-08T08:20:17.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.510+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c40b1af0 con 0x7ff5c415cc80 2023-11-08T08:20:17.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.511+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c40b20b0 con 0x7ff5c415cc80 2023-11-08T08:20:17.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.512+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5bc012050 con 0x7ff5c415cc80 2023-11-08T08:20:17.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.512+0000 7ff5c17fa700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682f0 0x7ff5a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.512+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff5bc0861e0 con 0x7ff5c415cc80 2023-11-08T08:20:17.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.512+0000 7ff5c8cfe700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682f0 0x7ff5a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.513+0000 7ff5c8cfe700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682f0 0x7ff5a806a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009c40 tx=0x7ff5b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.517+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5bc052a40 con 0x7ff5c415cc80 2023-11-08T08:20:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.699+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5c40743f0 con 0x7ff5c415cc80 2023-11-08T08:20:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.700+0000 7ff5c17fa700 1 -- 172.21.15.116:0/4196084138 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+17969 (secure 0 0 0) 0x7ff5bc056060 con 0x7ff5c415cc80 2023-11-08T08:20:17.702 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:17.702 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:46.066442+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682f0 msgr2=0x7ff5a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682f0 0x7ff5a806a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009c40 tx=0x7ff5b8008030 comp rx=0 tx=0).stop 2023-11-08T08:20:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 msgr2=0x7ff5c40b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 0x7ff5c40b4700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc007e90 tx=0x7ff5bc00e3c0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 shutdown_connections 2023-11-08T08:20:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c415cc80 0x7ff5c40b4700 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5c406a270 0x7ff5c414b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5a80682f0 0x7ff5a806a7a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.702+0000 7ff5caf62700 1 --2- 172.21.15.116:0/4196084138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5c406e570 0x7ff5c41e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.703+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 >> 172.21.15.116:0/4196084138 conn(0x7ff5c406c010 msgr2=0x7ff5c40ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.703+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 shutdown_connections 2023-11-08T08:20:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.703+0000 7ff5caf62700 1 -- 172.21.15.116:0/4196084138 wait complete. 2023-11-08T08:20:17.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:19:43.321551+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:19:44.023597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:20:17.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:20:17.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.838+0000 7fb3f481b700 1 Processor -- start 2023-11-08T08:20:17.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.838+0000 7fb3f481b700 1 -- start start 2023-11-08T08:20:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.838+0000 7fb3f481b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3f0158e10 0x7fb3f00cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.838+0000 7fb3f481b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f006cb50 0x7fb3f0070370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3f481b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f00ca140 con 0x7fb3f00ae850 2023-11-08T08:20:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3f481b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f006ac30 con 0x7fb3f0158e10 2023-11-08T08:20:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3f481b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00c9b80 con 0x7fb3f006cb50 2023-11-08T08:20:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3eed9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f006cb50 0x7fb3f0070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ee59c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3f0158e10 0x7fb3f00cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3edd9b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb3f00ae850 0x7fb3f006c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:37084/0 (socket says 172.21.15.116:37084) 2023-11-08T08:20:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3edd9b700 1 -- 172.21.15.116:0/819530470 learned_addr learned my addr 172.21.15.116:0/819530470 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ed59a700 1 -- 172.21.15.116:0/819530470 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839712427 0 0) 0x7fb3f00ca140 con 0x7fb3f00ae850 2023-11-08T08:20:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ed59a700 1 -- 172.21.15.116:0/819530470 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d80035b0 con 0x7fb3f00ae850 2023-11-08T08:20:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ee59c700 1 -- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f006cb50 msgr2=0x7fb3f0070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ee59c700 1 --2- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f006cb50 0x7fb3f0070370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ee59c700 1 -- 172.21.15.116:0/819530470 >> v1:172.21.15.129:6789/0 conn(0x7fb3f00ae850 legacy=0x7fb3f006c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3ee59c700 1 -- 172.21.15.116:0/819530470 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f00c9e60 con 0x7fb3f0158e10 2023-11-08T08:20:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.839+0000 7fb3eed9d700 1 --2- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f006cb50 0x7fb3f0070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3ee59c700 1 --2- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f0158e10 0x7fb3f00cb5b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009840 tx=0x7fb3e4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=699a668a091bf675 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3ed59a700 1 -- 172.21.15.116:0/819530470 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4011830 con 0x7fb3f0158e10 2023-11-08T08:20:17.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3ed59a700 1 -- 172.21.15.116:0/819530470 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4011990 con 0x7fb3f0158e10 2023-11-08T08:20:17.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3ed59a700 1 -- 172.21.15.116:0/819530470 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4011d00 con 0x7fb3f0158e10 2023-11-08T08:20:17.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 -- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f0158e10 msgr2=0x7fb3f00cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 --2- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f0158e10 0x7fb3f00cb5b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009840 tx=0x7fb3e4009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:17.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 -- 172.21.15.116:0/819530470 shutdown_connections 2023-11-08T08:20:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 --2- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f0158e10 0x7fb3f00cb5b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 --2- 172.21.15.116:0/819530470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f006cb50 0x7fb3f0070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 -- 172.21.15.116:0/819530470 >> 172.21.15.116:0/819530470 conn(0x7fb3f00cf400 msgr2=0x7fb3f00cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 -- 172.21.15.116:0/819530470 shutdown_connections 2023-11-08T08:20:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.840+0000 7fb3f481b700 1 -- 172.21.15.116:0/819530470 wait complete. 2023-11-08T08:20:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.842+0000 7fb3f481b700 1 Processor -- start 2023-11-08T08:20:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.842+0000 7fb3f481b700 1 -- start start 2023-11-08T08:20:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3f481b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f006cb50 0x7fb3f00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3f481b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 0x7fb3f0157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3f481b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f0158e10 0x7fb3f0151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3f481b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0157870 con 0x7fb3f00ae850 2023-11-08T08:20:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3f481b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f015bfd0 con 0x7fb3f006cb50 2023-11-08T08:20:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3f481b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0156850 con 0x7fb3f0158e10 2023-11-08T08:20:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 0x7fb3f0157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 0x7fb3f0157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56212/0 (socket says 172.21.15.116:56212) 2023-11-08T08:20:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 -- 172.21.15.116:0/1215864543 learned_addr learned my addr 172.21.15.116:0/1215864543 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3ee59c700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f006cb50 0x7fb3f00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3eed9d700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f0158e10 0x7fb3f0151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 -- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f0158e10 msgr2=0x7fb3f0151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f0158e10 0x7fb3f0151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 -- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f006cb50 msgr2=0x7fb3f00af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f006cb50 0x7fb3f00af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3edd9b700 1 -- 172.21.15.116:0/1215864543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e8009a70 con 0x7fb3f00ae850 2023-11-08T08:20:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.843+0000 7fb3eed9d700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f0158e10 0x7fb3f0151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.844+0000 7fb3edd9b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 0x7fb3f0157330 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0001eb0 tx=0x7fb3e000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.844+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0005d70 con 0x7fb3f00ae850 2023-11-08T08:20:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.844+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e00070d0 con 0x7fb3f00ae850 2023-11-08T08:20:17.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.844+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0007480 con 0x7fb3f00ae850 2023-11-08T08:20:17.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.844+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e40094a0 con 0x7fb3f00ae850 2023-11-08T08:20:17.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.844+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f006b100 con 0x7fb3f00ae850 2023-11-08T08:20:17.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.845+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f006ae20 con 0x7fb3f00ae850 2023-11-08T08:20:17.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.845+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb3e0019030 con 0x7fb3f00ae850 2023-11-08T08:20:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.846+0000 7fb3deffd700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3d00682f0 0x7fb3d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.846+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb3e0085130 con 0x7fb3f00ae850 2023-11-08T08:20:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.846+0000 7fb3ee59c700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3d00682f0 0x7fb3d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.846+0000 7fb3ee59c700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3d00682f0 0x7fb3d006a7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4000c00 tx=0x7fb3e401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:17.850+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3e00519d0 con 0x7fb3f00ae850 2023-11-08T08:20:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.028+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb3f006b620 con 0x7fb3f00ae850 2023-11-08T08:20:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.028+0000 7fb3deffd700 1 -- 172.21.15.116:0/1215864543 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb3e0017020 con 0x7fb3f00ae850 2023-11-08T08:20:18.030 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:20:18.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3d00682f0 msgr2=0x7fb3d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3d00682f0 0x7fb3d006a7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4000c00 tx=0x7fb3e401c040 comp rx=0 tx=0).stop 2023-11-08T08:20:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 msgr2=0x7fb3f0157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 0x7fb3f0157330 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0001eb0 tx=0x7fb3e000c860 comp rx=0 tx=0).stop 2023-11-08T08:20:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 shutdown_connections 2023-11-08T08:20:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3f0158e10 0x7fb3f0151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3d00682f0 0x7fb3d006a7a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f006cb50 0x7fb3f00af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 --2- 172.21.15.116:0/1215864543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3f00ae850 0x7fb3f0157330 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 >> 172.21.15.116:0/1215864543 conn(0x7fb3f00cf400 msgr2=0x7fb3f00cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 shutdown_connections 2023-11-08T08:20:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.030+0000 7fb3f481b700 1 -- 172.21.15.116:0/1215864543 wait complete. 2023-11-08T08:20:18.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:20:18.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.165+0000 7fc52c5b8700 1 Processor -- start 2023-11-08T08:20:18.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.165+0000 7fc52c5b8700 1 -- start start 2023-11-08T08:20:18.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.165+0000 7fc52c5b8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc5240cf710 0x7fc5240ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.165+0000 7fc52c5b8700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52415cc10 con 0x7fc52406a980 2023-11-08T08:20:18.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.165+0000 7fc52c5b8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52406c9b0 con 0x7fc524071890 2023-11-08T08:20:18.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.165+0000 7fc52c5b8700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52406cc90 con 0x7fc5240cf710 2023-11-08T08:20:18.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52a354700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc5240cf710 0x7fc5240ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52ab55700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc52406a980 0x7fc524160480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58388/0 (socket says 172.21.15.116:58388) 2023-11-08T08:20:18.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52ab55700 1 -- 172.21.15.116:0/2347898543 learned_addr learned my addr 172.21.15.116:0/2347898543 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:18.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52a354700 1 -- 172.21.15.116:0/2347898543 >> v1:172.21.15.129:6789/0 conn(0x7fc524071890 legacy=0x7fc52406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:18.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52a354700 1 -- 172.21.15.116:0/2347898543 >> v1:172.21.15.116:6789/0 conn(0x7fc52406a980 legacy=0x7fc524160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52a354700 1 -- 172.21.15.116:0/2347898543 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52406cf70 con 0x7fc5240cf710 2023-11-08T08:20:18.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc52a354700 1 --2- 172.21.15.116:0/2347898543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240ce740 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc514009870 tx=0x7fc514009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e76212e81222ce7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc529352700 1 -- 172.21.15.116:0/2347898543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514004230 con 0x7fc5240cf710 2023-11-08T08:20:18.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc529352700 1 -- 172.21.15.116:0/2347898543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc514004390 con 0x7fc5240cf710 2023-11-08T08:20:18.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.166+0000 7fc529352700 1 -- 172.21.15.116:0/2347898543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5140036a0 con 0x7fc5240cf710 2023-11-08T08:20:18.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 -- 172.21.15.116:0/2347898543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 msgr2=0x7fc5240ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/2347898543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240ce740 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc514009870 tx=0x7fc514009b80 comp rx=0 tx=0).stop 2023-11-08T08:20:18.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 -- 172.21.15.116:0/2347898543 shutdown_connections 2023-11-08T08:20:18.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/2347898543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240ce740 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 -- 172.21.15.116:0/2347898543 >> 172.21.15.116:0/2347898543 conn(0x7fc52406c010 msgr2=0x7fc5240cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:18.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 -- 172.21.15.116:0/2347898543 shutdown_connections 2023-11-08T08:20:18.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.167+0000 7fc52c5b8700 1 -- 172.21.15.116:0/2347898543 wait complete. 2023-11-08T08:20:18.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.168+0000 7fc52c5b8700 1 Processor -- start 2023-11-08T08:20:18.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.168+0000 7fc52c5b8700 1 -- start start 2023-11-08T08:20:18.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc52406a980 0x7fc5240cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc524071890 0x7fc52414db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52414c010 con 0x7fc5240cf710 2023-11-08T08:20:18.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240cb170 con 0x7fc52406a980 2023-11-08T08:20:18.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52414eb10 con 0x7fc524071890 2023-11-08T08:20:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56244/0 (socket says 172.21.15.116:56244) 2023-11-08T08:20:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 -- 172.21.15.116:0/3288081241 learned_addr learned my addr 172.21.15.116:0/3288081241 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc529b53700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc524071890 0x7fc52414db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52a354700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc52406a980 0x7fc5240cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 -- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc524071890 msgr2=0x7fc52414db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc524071890 0x7fc52414db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 -- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc52406a980 msgr2=0x7fc5240cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc52406a980 0x7fc5240cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 -- 172.21.15.116:0/3288081241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140094a0 con 0x7fc5240cf710 2023-11-08T08:20:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52a354700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc52406a980 0x7fc5240cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc529b53700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc524071890 0x7fc52414db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52ab55700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240b0af0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00caf0 tx=0x7fc51c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c00aad0 con 0x7fc5240cf710 2023-11-08T08:20:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c00ac30 con 0x7fc5240cf710 2023-11-08T08:20:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52406d530 con 0x7fc5240cf710 2023-11-08T08:20:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c012400 con 0x7fc5240cf710 2023-11-08T08:20:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.169+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52406d250 con 0x7fc5240cf710 2023-11-08T08:20:18.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.170+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52414c150 con 0x7fc5240cf710 2023-11-08T08:20:18.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.171+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc51c019030 con 0x7fc5240cf710 2023-11-08T08:20:18.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.171+0000 7fc51affd700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc5080682f0 0x7fc50806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.171+0000 7fc52a354700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc5080682f0 0x7fc50806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.171+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc51c085770 con 0x7fc5240cf710 2023-11-08T08:20:18.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.172+0000 7fc52a354700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc5080682f0 0x7fc50806a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc514009840 tx=0x7fc514008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.175+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc51c04b5e0 con 0x7fc5240cf710 2023-11-08T08:20:18.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.354+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5240743f0 con 0x7fc5240cf710 2023-11-08T08:20:18.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.355+0000 7fc51affd700 1 -- 172.21.15.116:0/3288081241 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc52414c150 con 0x7fc5240cf710 2023-11-08T08:20:18.356 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:20:18.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc5080682f0 msgr2=0x7fc50806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc5080682f0 0x7fc50806a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc514009840 tx=0x7fc514008030 comp rx=0 tx=0).stop 2023-11-08T08:20:18.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 msgr2=0x7fc5240b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240b0af0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00caf0 tx=0x7fc51c00ceb0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 shutdown_connections 2023-11-08T08:20:18.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5240cf710 0x7fc5240b0af0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc5080682f0 0x7fc50806a7a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc52406a980 0x7fc5240cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 --2- 172.21.15.116:0/3288081241 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc524071890 0x7fc52414db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 >> 172.21.15.116:0/3288081241 conn(0x7fc52406c010 msgr2=0x7fc5240adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:18.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 shutdown_connections 2023-11-08T08:20:18.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.357+0000 7fc52c5b8700 1 -- 172.21.15.116:0/3288081241 wait complete. 2023-11-08T08:20:18.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:20:18.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.491+0000 7f1f4b903700 1 Processor -- start 2023-11-08T08:20:18.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.491+0000 7f1f4b903700 1 -- start start 2023-11-08T08:20:18.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4b903700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f4415c8c0 0x7f1f44160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4b903700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1f440b5dc0 0x7f1f4406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4b903700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f4406b050 0x7f1f440d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4b903700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4406b4d0 con 0x7f1f4415c8c0 2023-11-08T08:20:18.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4b903700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f440d2d30 con 0x7f1f440b5dc0 2023-11-08T08:20:18.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4b903700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440d0b40 con 0x7f1f4406b050 2023-11-08T08:20:18.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4969f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f4415c8c0 0x7f1f44160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4969f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f4415c8c0 0x7f1f44160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56250/0 (socket says 172.21.15.116:56250) 2023-11-08T08:20:18.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.492+0000 7f1f4969f700 1 -- 172.21.15.116:0/3731211482 learned_addr learned my addr 172.21.15.116:0/3731211482 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:18.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f48e9e700 1 --2- 172.21.15.116:0/3731211482 >> v2:172.21.15.129:3300/0 conn(0x7f1f440b5dc0 0x7f1f4406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f49ea0700 1 --2- 172.21.15.116:0/3731211482 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f4406b050 0x7f1f440d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4969f700 1 -- 172.21.15.116:0/3731211482 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f4406b050 msgr2=0x7f1f440d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:18.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4969f700 1 --2- 172.21.15.116:0/3731211482 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f4406b050 0x7f1f440d0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4969f700 1 -- 172.21.15.116:0/3731211482 >> v2:172.21.15.129:3300/0 conn(0x7f1f440b5dc0 msgr2=0x7f1f4406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:18.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4969f700 1 --2- 172.21.15.116:0/3731211482 >> v2:172.21.15.129:3300/0 conn(0x7f1f440b5dc0 0x7f1f4406ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4969f700 1 -- 172.21.15.116:0/3731211482 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f441611f0 con 0x7f1f4415c8c0 2023-11-08T08:20:18.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4969f700 1 --2- 172.21.15.116:0/3731211482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4415c8c0 0x7f1f44160cb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1f440d24c0 tx=0x7f1f34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ff0efc8e67363a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f3bfff700 1 -- 172.21.15.116:0/3731211482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400b870 con 0x7f1f4415c8c0 2023-11-08T08:20:18.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f3bfff700 1 -- 172.21.15.116:0/3731211482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3400b9d0 con 0x7f1f4415c8c0 2023-11-08T08:20:18.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f3bfff700 1 -- 172.21.15.116:0/3731211482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400bd60 con 0x7f1f4415c8c0 2023-11-08T08:20:18.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4b903700 1 -- 172.21.15.116:0/3731211482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4415c8c0 msgr2=0x7f1f44160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.493+0000 7f1f4b903700 1 --2- 172.21.15.116:0/3731211482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4415c8c0 0x7f1f44160cb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1f440d24c0 tx=0x7f1f34009bd0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 -- 172.21.15.116:0/3731211482 shutdown_connections 2023-11-08T08:20:18.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 --2- 172.21.15.116:0/3731211482 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f4406b050 0x7f1f440d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 --2- 172.21.15.116:0/3731211482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4415c8c0 0x7f1f44160cb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 --2- 172.21.15.116:0/3731211482 >> v2:172.21.15.129:3300/0 conn(0x7f1f440b5dc0 0x7f1f4406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 -- 172.21.15.116:0/3731211482 >> 172.21.15.116:0/3731211482 conn(0x7f1f4406c520 msgr2=0x7f1f4406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:18.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 -- 172.21.15.116:0/3731211482 shutdown_connections 2023-11-08T08:20:18.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.494+0000 7f1f4b903700 1 -- 172.21.15.116:0/3731211482 wait complete. 2023-11-08T08:20:18.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.495+0000 7f1f4b903700 1 Processor -- start 2023-11-08T08:20:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.495+0000 7f1f4b903700 1 -- start start 2023-11-08T08:20:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.495+0000 7f1f4b903700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 0x7f1f44155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f440b5dc0 0x7f1f4415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f4415c8c0 0x7f1f44157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4415fc80 con 0x7f1f4406b050 2023-11-08T08:20:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4415d3b0 con 0x7f1f4415c8c0 2023-11-08T08:20:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440cb610 con 0x7f1f440b5dc0 2023-11-08T08:20:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 0x7f1f44155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 0x7f1f44155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56258/0 (socket says 172.21.15.116:56258) 2023-11-08T08:20:18.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 -- 172.21.15.116:0/2125095808 learned_addr learned my addr 172.21.15.116:0/2125095808 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:18.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f48e9e700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f440b5dc0 0x7f1f4415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f49ea0700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f4415c8c0 0x7f1f44157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 -- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f440b5dc0 msgr2=0x7f1f4415f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f440b5dc0 0x7f1f4415f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 -- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f4415c8c0 msgr2=0x7f1f44157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f4415c8c0 0x7f1f44157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 -- 172.21.15.116:0/2125095808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f34009550 con 0x7f1f4406b050 2023-11-08T08:20:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4969f700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 0x7f1f44155ff0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f34000c00 tx=0x7f1f34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34004310 con 0x7f1f4406b050 2023-11-08T08:20:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3401e070 con 0x7f1f4406b050 2023-11-08T08:20:18.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f440cd760 con 0x7f1f4406b050 2023-11-08T08:20:18.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f340036a0 con 0x7f1f4406b050 2023-11-08T08:20:18.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.496+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f440d2a50 con 0x7f1f4406b050 2023-11-08T08:20:18.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.497+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f440cda40 con 0x7f1f4406b050 2023-11-08T08:20:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.498+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f34029020 con 0x7f1f4406b050 2023-11-08T08:20:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.498+0000 7f1f39ffb700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f280682f0 0x7f1f2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.498+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1f34086ca0 con 0x7f1f4406b050 2023-11-08T08:20:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.498+0000 7f1f48e9e700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f280682f0 0x7f1f2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.498+0000 7f1f48e9e700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f280682f0 0x7f1f2806a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1f440cd140 tx=0x7f1f40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.503+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f3408b070 con 0x7f1f4406b050 2023-11-08T08:20:18.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.681+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f44067d70 con 0x7f1f4406b050 2023-11-08T08:20:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.681+0000 7f1f39ffb700 1 -- 172.21.15.116:0/2125095808 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f34053500 con 0x7f1f4406b050 2023-11-08T08:20:18.683 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:20:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f280682f0 msgr2=0x7f1f2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f280682f0 0x7f1f2806a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1f440cd140 tx=0x7f1f40008040 comp rx=0 tx=0).stop 2023-11-08T08:20:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 msgr2=0x7f1f44155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 0x7f1f44155ff0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f34000c00 tx=0x7f1f34004750 comp rx=0 tx=0).stop 2023-11-08T08:20:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 shutdown_connections 2023-11-08T08:20:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f280682f0 0x7f1f2806a7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f4415c8c0 0x7f1f44157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4406b050 0x7f1f44155ff0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 --2- 172.21.15.116:0/2125095808 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f440b5dc0 0x7f1f4415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 >> 172.21.15.116:0/2125095808 conn(0x7f1f4406c520 msgr2=0x7f1f440b8930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:18.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.683+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 shutdown_connections 2023-11-08T08:20:18.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.684+0000 7f1f4b903700 1 -- 172.21.15.116:0/2125095808 wait complete. 2023-11-08T08:20:18.696 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:20:18.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.818+0000 7f54eff2e700 1 Processor -- start 2023-11-08T08:20:18.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.818+0000 7f54eff2e700 1 -- start start 2023-11-08T08:20:18.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54eff2e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54e80b2ee0 0x7f54e80d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54eff2e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80cc200 0x7f54e80ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54eff2e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e815a9c0 con 0x7f54e815a5f0 2023-11-08T08:20:18.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54eff2e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e80c7b80 con 0x7f54e80b2ee0 2023-11-08T08:20:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54eff2e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80cc990 con 0x7f54e80cc200 2023-11-08T08:20:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54e80b2ee0 0x7f54e80d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54e80b2ee0 0x7f54e80d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56274/0 (socket says 172.21.15.116:56274) 2023-11-08T08:20:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 -- 172.21.15.116:0/3415154027 learned_addr learned my addr 172.21.15.116:0/3415154027 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 -- 172.21.15.116:0/3415154027 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80cc200 msgr2=0x7f54e80ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 --2- 172.21.15.116:0/3415154027 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80cc200 0x7f54e80ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 -- 172.21.15.116:0/3415154027 >> v1:172.21.15.129:6789/0 conn(0x7f54e815a5f0 legacy=0x7f54e80b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.819+0000 7f54ed4c9700 1 -- 172.21.15.116:0/3415154027 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80ccc70 con 0x7f54e80b2ee0 2023-11-08T08:20:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54ed4c9700 1 --2- 172.21.15.116:0/3415154027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e80b2ee0 0x7f54e80d0ad0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f54e40098f0 tx=0x7f54e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1f4197099af2d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eccc8700 1 -- 172.21.15.116:0/3415154027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4004230 con 0x7f54e80b2ee0 2023-11-08T08:20:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eccc8700 1 -- 172.21.15.116:0/3415154027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e4004390 con 0x7f54e80b2ee0 2023-11-08T08:20:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eccc8700 1 -- 172.21.15.116:0/3415154027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e40036a0 con 0x7f54e80b2ee0 2023-11-08T08:20:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 -- 172.21.15.116:0/3415154027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e80b2ee0 msgr2=0x7f54e80d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3415154027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e80b2ee0 0x7f54e80d0ad0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f54e40098f0 tx=0x7f54e4009c00 comp rx=0 tx=0).stop 2023-11-08T08:20:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 -- 172.21.15.116:0/3415154027 shutdown_connections 2023-11-08T08:20:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3415154027 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80cc200 0x7f54e80ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3415154027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e80b2ee0 0x7f54e80d0ad0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 -- 172.21.15.116:0/3415154027 >> 172.21.15.116:0/3415154027 conn(0x7f54e806ea80 msgr2=0x7f54e806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:18.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 -- 172.21.15.116:0/3415154027 shutdown_connections 2023-11-08T08:20:18.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.820+0000 7f54eff2e700 1 -- 172.21.15.116:0/3415154027 wait complete. 2023-11-08T08:20:18.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.821+0000 7f54eff2e700 1 Processor -- start 2023-11-08T08:20:18.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 -- start start 2023-11-08T08:20:18.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80b2ee0 0x7f54e80c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e80cc200 0x7f54e814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 0x7f54e80ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e814b030 con 0x7f54e815a5f0 2023-11-08T08:20:18.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80b4630 con 0x7f54e80cc200 2023-11-08T08:20:18.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54eff2e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80b3d00 con 0x7f54e80b2ee0 2023-11-08T08:20:18.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54ee4cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 0x7f54e80ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54ee4cb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 0x7f54e80ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56284/0 (socket says 172.21.15.116:56284) 2023-11-08T08:20:18.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.822+0000 7f54ee4cb700 1 -- 172.21.15.116:0/3095372539 learned_addr learned my addr 172.21.15.116:0/3095372539 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:18.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ed4c9700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e80cc200 0x7f54e814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54edcca700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80b2ee0 0x7f54e80c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ee4cb700 1 -- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80b2ee0 msgr2=0x7f54e80c48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ee4cb700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80b2ee0 0x7f54e80c48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ee4cb700 1 -- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e80cc200 msgr2=0x7f54e814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:18.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ee4cb700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e80cc200 0x7f54e814aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:18.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ee4cb700 1 -- 172.21.15.116:0/3095372539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e4009550 con 0x7f54e815a5f0 2023-11-08T08:20:18.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ed4c9700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e80cc200 0x7f54e814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54ee4cb700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 0x7f54e80ced30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f54e000b9a0 tx=0x7f54e000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e000c9b0 con 0x7f54e815a5f0 2023-11-08T08:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e000cb30 con 0x7f54e815a5f0 2023-11-08T08:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e80cd230 con 0x7f54e815a5f0 2023-11-08T08:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e0014440 con 0x7f54e815a5f0 2023-11-08T08:20:18.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.823+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e80ccf50 con 0x7f54e815a5f0 2023-11-08T08:20:18.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.824+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e80cd510 con 0x7f54e815a5f0 2023-11-08T08:20:18.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.824+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f54e001b030 con 0x7f54e815a5f0 2023-11-08T08:20:18.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.825+0000 7f54de7fc700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f54cc068320 0x7f54cc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:18.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.825+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f54e0006e40 con 0x7f54e815a5f0 2023-11-08T08:20:18.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.825+0000 7f54edcca700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f54cc068320 0x7f54cc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:18.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.825+0000 7f54edcca700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f54cc068320 0x7f54cc06a7d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f54d8001670 tx=0x7f54d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:18.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:18.829+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f54e0052180 con 0x7f54e815a5f0 2023-11-08T08:20:18.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:20:18.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:20:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.012+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f54e8155e60 con 0x7f54e815a5f0 2023-11-08T08:20:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.013+0000 7f54de7fc700 1 -- 172.21.15.116:0/3095372539 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+11 (secure 0 0 0) 0x7f54e0019020 con 0x7f54e815a5f0 2023-11-08T08:20:19.014 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:20:19.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f54cc068320 msgr2=0x7f54cc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f54cc068320 0x7f54cc06a7d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f54d8001670 tx=0x7f54d8009040 comp rx=0 tx=0).stop 2023-11-08T08:20:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 msgr2=0x7f54e80ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 0x7f54e80ced30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f54e000b9a0 tx=0x7f54e000bd60 comp rx=0 tx=0).stop 2023-11-08T08:20:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 shutdown_connections 2023-11-08T08:20:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e815a5f0 0x7f54e80ced30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f54e80b2ee0 0x7f54e80c48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f54cc068320 0x7f54cc06a7d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 --2- 172.21.15.116:0/3095372539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e80cc200 0x7f54e814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 >> 172.21.15.116:0/3095372539 conn(0x7f54e806ea80 msgr2=0x7f54e81e3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:19.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 shutdown_connections 2023-11-08T08:20:19.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.015+0000 7f54eff2e700 1 -- 172.21.15.116:0/3095372539 wait complete. 2023-11-08T08:20:19.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff50da92700 1 Processor -- start 2023-11-08T08:20:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff50da92700 1 -- start start 2023-11-08T08:20:19.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff50da92700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff5080b22a0 0x7ff5080cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff50da92700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080d1ee0 con 0x7ff50806ab10 2023-11-08T08:20:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff50da92700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080cdb10 con 0x7ff5080cd5a0 2023-11-08T08:20:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff50da92700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5080d1890 con 0x7ff5080b22a0 2023-11-08T08:20:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.193+0000 7ff5077fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff50806ab10 0x7ff50806aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58398/0 (socket says 172.21.15.116:58398) 2023-11-08T08:20:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5077fe700 1 -- 172.21.15.116:0/387826714 learned_addr learned my addr 172.21.15.116:0/387826714 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743066512 0 0) 0x7ff5080d1ee0 con 0x7ff50806ab10 2023-11-08T08:20:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 --2- 172.21.15.116:0/387826714 >> v2:172.21.15.129:3300/0 conn(0x7ff5080b22a0 0x7ff5080cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f40035b0 con 0x7ff50806ab10 2023-11-08T08:20:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1942559777 0 0) 0x7ff4f40035b0 con 0x7ff50806ab10 2023-11-08T08:20:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 >> v2:172.21.15.129:3300/0 conn(0x7ff5080b22a0 msgr2=0x7ff5080cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 --2- 172.21.15.116:0/387826714 >> v2:172.21.15.129:3300/0 conn(0x7ff5080b22a0 0x7ff5080cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 >> v1:172.21.15.129:6789/0 conn(0x7ff5080cd5a0 legacy=0x7ff5080cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080d1b70 con 0x7ff50806ab10 2023-11-08T08:20:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7ff4f8001670 con 0x7ff50806ab10 2023-11-08T08:20:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 >> v1:172.21.15.116:6789/0 conn(0x7ff50806ab10 legacy=0x7ff50806aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4003a00 0x7ff4f4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4f4004850 0x7ff4f4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4f4005200 0x7ff4f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080d1890 con 0x7ff4f4005200 2023-11-08T08:20:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f400c080 con 0x7ff4f4003a00 2023-11-08T08:20:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f400c240 con 0x7ff4f4004850 2023-11-08T08:20:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4f4005200 0x7ff4f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff5077fe700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4003a00 0x7ff4f4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff506ffd700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4f4004850 0x7ff4f4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 -- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4f4004850 msgr2=0x7ff4f4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4f4004850 0x7ff4f4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 -- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4003a00 msgr2=0x7ff4f4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4003a00 0x7ff4f4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.194+0000 7ff507fff700 1 -- 172.21.15.116:0/387826714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f40035b0 con 0x7ff4f4005200 2023-11-08T08:20:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff507fff700 1 -- 172.21.15.116:0/387826714 reap_dead start 2023-11-08T08:20:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff507fff700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4f4005200 0x7ff4f400ba10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff500009fc0 tx=0x7ff4f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73dc284b26764a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500003a20 con 0x7ff4f4005200 2023-11-08T08:20:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50000f040 con 0x7ff4f4005200 2023-11-08T08:20:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff5067fc700 1 -- 172.21.15.116:0/387826714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500003de0 con 0x7ff4f4005200 2023-11-08T08:20:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 -- 172.21.15.116:0/387826714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4f4005200 msgr2=0x7ff4f400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4f4005200 0x7ff4f400ba10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff500009fc0 tx=0x7ff4f4005880 comp rx=0 tx=0).stop 2023-11-08T08:20:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 -- 172.21.15.116:0/387826714 shutdown_connections 2023-11-08T08:20:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 --2- 172.21.15.116:0/387826714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4f4005200 0x7ff4f400ba10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 -- 172.21.15.116:0/387826714 >> 172.21.15.116:0/387826714 conn(0x7ff50806c520 msgr2=0x7ff50806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 -- 172.21.15.116:0/387826714 shutdown_connections 2023-11-08T08:20:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.195+0000 7ff50da92700 1 -- 172.21.15.116:0/387826714 wait complete. 2023-11-08T08:20:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.196+0000 7ff50da92700 1 Processor -- start 2023-11-08T08:20:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 -- start start 2023-11-08T08:20:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4005200 0x7ff5080cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 0x7ff5080b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b84c0 0x7ff5080c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50815e360 con 0x7ff508154500 2023-11-08T08:20:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080cb660 con 0x7ff4f4005200 2023-11-08T08:20:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.197+0000 7ff50da92700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080b8930 con 0x7ff5080b84c0 2023-11-08T08:20:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 0x7ff5080b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 0x7ff5080b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56300/0 (socket says 172.21.15.116:56300) 2023-11-08T08:20:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 -- 172.21.15.116:0/3456023812 learned_addr learned my addr 172.21.15.116:0/3456023812 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff5077fe700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4005200 0x7ff5080cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff507fff700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b84c0 0x7ff5080c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 -- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b84c0 msgr2=0x7ff5080c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b84c0 0x7ff5080c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 -- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4005200 msgr2=0x7ff5080cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4005200 0x7ff5080cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 -- 172.21.15.116:0/3456023812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080b1ec0 con 0x7ff508154500 2023-11-08T08:20:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff506ffd700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 0x7ff5080b7f80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff4f000e4c0 tx=0x7ff4f000e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f000ee50 con 0x7ff508154500 2023-11-08T08:20:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f00052e0 con 0x7ff508154500 2023-11-08T08:20:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50815cad0 con 0x7ff508154500 2023-11-08T08:20:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0003680 con 0x7ff508154500 2023-11-08T08:20:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.198+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080b2050 con 0x7ff508154500 2023-11-08T08:20:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.199+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5080743f0 con 0x7ff508154500 2023-11-08T08:20:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.199+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4f001b030 con 0x7ff508154500 2023-11-08T08:20:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.200+0000 7ff4fffff700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4e80682f0 0x7ff4e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.200+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff4f0084fe0 con 0x7ff508154500 2023-11-08T08:20:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.200+0000 7ff5077fe700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4e80682f0 0x7ff4e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.200+0000 7ff5077fe700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4e80682f0 0x7ff4e806a7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8001c90 tx=0x7ff4f8000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.204+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4f004d5e0 con 0x7ff508154500 2023-11-08T08:20:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.425+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5080d2c80 con 0x7ff508154500 2023-11-08T08:20:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.426+0000 7ff4fffff700 1 -- 172.21.15.116:0/3456023812 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7ff4f0019090 con 0x7ff508154500 2023-11-08T08:20:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:19.427 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:20:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.427+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4e80682f0 msgr2=0x7ff4e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.427+0000 7ff50da92700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4e80682f0 0x7ff4e806a7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8001c90 tx=0x7ff4f8000ed0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.427+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 msgr2=0x7ff5080b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.427+0000 7ff50da92700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 0x7ff5080b7f80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff4f000e4c0 tx=0x7ff4f000e7d0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 shutdown_connections 2023-11-08T08:20:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b84c0 0x7ff5080c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4e80682f0 0x7ff4e806a7a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f4005200 0x7ff5080cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 --2- 172.21.15.116:0/3456023812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff508154500 0x7ff5080b7f80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 >> 172.21.15.116:0/3456023812 conn(0x7ff50806c520 msgr2=0x7ff5081557e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 shutdown_connections 2023-11-08T08:20:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.428+0000 7ff50da92700 1 -- 172.21.15.116:0/3456023812 wait complete. 2023-11-08T08:20:19.431 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:20:19.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:19.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.560+0000 7f6e5299e700 1 Processor -- start 2023-11-08T08:20:19.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.560+0000 7f6e5299e700 1 -- start start 2023-11-08T08:20:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5299e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c1593b0 0x7f6e4c0cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5299e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e4c06ba90 0x7f6e4c0b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5299e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c0cdeb0 con 0x7f6e4c0ae200 2023-11-08T08:20:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5299e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0ae7c0 con 0x7f6e4c06ba90 2023-11-08T08:20:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5299e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0ce150 con 0x7f6e4c1593b0 2023-11-08T08:20:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5219d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e4c06ba90 0x7f6e4c0b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5119b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6e4c0ae200 0x7f6e4c06b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58410/0 (socket says 172.21.15.116:58410) 2023-11-08T08:20:19.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5119b700 1 -- 172.21.15.116:0/611369858 learned_addr learned my addr 172.21.15.116:0/611369858 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:19.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.561+0000 7f6e5199c700 1 --2- 172.21.15.116:0/611369858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c1593b0 0x7f6e4c0cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5219d700 1 -- 172.21.15.116:0/611369858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c1593b0 msgr2=0x7f6e4c0cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:19.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5219d700 1 --2- 172.21.15.116:0/611369858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c1593b0 0x7f6e4c0cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5219d700 1 -- 172.21.15.116:0/611369858 >> v1:172.21.15.116:6789/0 conn(0x7f6e4c0ae200 legacy=0x7f6e4c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5099a700 1 -- 172.21.15.116:0/611369858 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927539383 0 0) 0x7f6e4c0cdeb0 con 0x7f6e4c0ae200 2023-11-08T08:20:19.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5219d700 1 -- 172.21.15.116:0/611369858 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0ce430 con 0x7f6e4c06ba90 2023-11-08T08:20:19.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5219d700 1 --2- 172.21.15.116:0/611369858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c0b3070 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009d90 tx=0x7f6e4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d90c13b7ed27d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5099a700 1 -- 172.21.15.116:0/611369858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400e070 con 0x7f6e4c06ba90 2023-11-08T08:20:19.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5099a700 1 -- 172.21.15.116:0/611369858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4400bbd0 con 0x7f6e4c06ba90 2023-11-08T08:20:19.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5099a700 1 -- 172.21.15.116:0/611369858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44012430 con 0x7f6e4c06ba90 2023-11-08T08:20:19.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 -- 172.21.15.116:0/611369858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 msgr2=0x7f6e4c0b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 --2- 172.21.15.116:0/611369858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c0b3070 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009d90 tx=0x7f6e4400b330 comp rx=0 tx=0).stop 2023-11-08T08:20:19.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 -- 172.21.15.116:0/611369858 shutdown_connections 2023-11-08T08:20:19.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 --2- 172.21.15.116:0/611369858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c0b3070 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 --2- 172.21.15.116:0/611369858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c1593b0 0x7f6e4c0cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 -- 172.21.15.116:0/611369858 >> 172.21.15.116:0/611369858 conn(0x7f6e4c0cc950 msgr2=0x7f6e4c0cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:19.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 -- 172.21.15.116:0/611369858 shutdown_connections 2023-11-08T08:20:19.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.562+0000 7f6e5299e700 1 -- 172.21.15.116:0/611369858 wait complete. 2023-11-08T08:20:19.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.564+0000 7f6e5299e700 1 Processor -- start 2023-11-08T08:20:19.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.564+0000 7f6e5299e700 1 -- start start 2023-11-08T08:20:19.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5299e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c09aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5299e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c0ae200 0x7f6e4c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5299e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e4c1593b0 0x7f6e4c157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5299e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c09cb40 con 0x7f6e4c06ba90 2023-11-08T08:20:19.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5299e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c157680 con 0x7f6e4c1593b0 2023-11-08T08:20:19.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5299e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0b00c0 con 0x7f6e4c0ae200 2023-11-08T08:20:19.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c09aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c09aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56314/0 (socket says 172.21.15.116:56314) 2023-11-08T08:20:19.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 -- 172.21.15.116:0/2101030337 learned_addr learned my addr 172.21.15.116:0/2101030337 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:19.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 -- 172.21.15.116:0/2101030337 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c0ae200 msgr2=0x7f6e4c157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c0ae200 0x7f6e4c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 -- 172.21.15.116:0/2101030337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e4c1593b0 msgr2=0x7f6e4c157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e4c1593b0 0x7f6e4c157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.565+0000 7f6e5199c700 1 -- 172.21.15.116:0/2101030337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44009a70 con 0x7f6e4c06ba90 2023-11-08T08:20:19.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.566+0000 7f6e5199c700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c09aef0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00c820 tx=0x7f6e3c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.566+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c011800 con 0x7f6e4c06ba90 2023-11-08T08:20:19.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.566+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c011960 con 0x7f6e4c06ba90 2023-11-08T08:20:19.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.566+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c159780 con 0x7f6e4c06ba90 2023-11-08T08:20:19.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.566+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c011cf0 con 0x7f6e4c06ba90 2023-11-08T08:20:19.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.566+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c0aeaa0 con 0x7f6e4c06ba90 2023-11-08T08:20:19.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.567+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c0ce950 con 0x7f6e4c06ba90 2023-11-08T08:20:19.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.567+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6e3c021070 con 0x7f6e4c06ba90 2023-11-08T08:20:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.567+0000 7f6e427fc700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6e300682f0 0x7f6e3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.567+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6e3c085a20 con 0x7f6e4c06ba90 2023-11-08T08:20:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.567+0000 7f6e5119b700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6e300682f0 0x7f6e3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.568+0000 7f6e5119b700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6e300682f0 0x7f6e3006a7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6e480011e0 tx=0x7f6e480079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.572+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6e3c052280 con 0x7f6e4c06ba90 2023-11-08T08:20:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.753+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e4c0725e0 con 0x7f6e4c06ba90 2023-11-08T08:20:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.754+0000 7f6e427fc700 1 -- 172.21.15.116:0/2101030337 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+17969 (secure 0 0 0) 0x7f6e3c0558a0 con 0x7f6e4c06ba90 2023-11-08T08:20:19.755 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:19.755 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:46.066442+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6e300682f0 msgr2=0x7f6e3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6e300682f0 0x7f6e3006a7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6e480011e0 tx=0x7f6e480079b0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 msgr2=0x7f6e4c09aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c09aef0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00c820 tx=0x7f6e3c00cbe0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 shutdown_connections 2023-11-08T08:20:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6e300682f0 0x7f6e3006a7a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e4c1593b0 0x7f6e4c157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e4c06ba90 0x7f6e4c09aef0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 --2- 172.21.15.116:0/2101030337 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6e4c0ae200 0x7f6e4c157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 >> 172.21.15.116:0/2101030337 conn(0x7f6e4c0cc950 msgr2=0x7f6e4c1589a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:19.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 shutdown_connections 2023-11-08T08:20:19.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.756+0000 7f6e5299e700 1 -- 172.21.15.116:0/2101030337 wait complete. 2023-11-08T08:20:19.769 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-08T08:20:19.769 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:19.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.888+0000 7f81154a0700 1 Processor -- start 2023-11-08T08:20:19.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.888+0000 7f81154a0700 1 -- start start 2023-11-08T08:20:19.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f81154a0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f81100b4660 0x7f8110071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f81154a0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81100b4a30 0x7f81100b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f81154a0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100cfb00 con 0x7f81100cff90 2023-11-08T08:20:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f81154a0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811015aa20 con 0x7f81100b4a30 2023-11-08T08:20:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f81154a0700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81100cf4d0 con 0x7f81100b4660 2023-11-08T08:20:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f810f7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81100b4a30 0x7f81100b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f810f7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81100b4a30 0x7f81100b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56322/0 (socket says 172.21.15.116:56322) 2023-11-08T08:20:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.889+0000 7f810f7fe700 1 -- 172.21.15.116:0/3736152313 learned_addr learned my addr 172.21.15.116:0/3736152313 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810f7fe700 1 -- 172.21.15.116:0/3736152313 >> v2:172.21.15.129:3300/0 conn(0x7f81100b4660 msgr2=0x7f8110071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810effd700 1 --2- 172.21.15.116:0/3736152313 >> v2:172.21.15.129:3300/0 conn(0x7f81100b4660 0x7f8110071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810f7fe700 1 --2- 172.21.15.116:0/3736152313 >> v2:172.21.15.129:3300/0 conn(0x7f81100b4660 0x7f8110071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810f7fe700 1 -- 172.21.15.116:0/3736152313 >> v1:172.21.15.129:6789/0 conn(0x7f81100cff90 legacy=0x7f811006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810f7fe700 1 -- 172.21.15.116:0/3736152313 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110072550 con 0x7f81100b4a30 2023-11-08T08:20:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810f7fe700 1 --2- 172.21.15.116:0/3736152313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4a30 0x7f81100b4e90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f811006efd0 tx=0x7f810800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e40928d89b5d7b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810dffb700 1 -- 172.21.15.116:0/3736152313 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810800e070 con 0x7f81100b4a30 2023-11-08T08:20:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810dffb700 1 -- 172.21.15.116:0/3736152313 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108003a20 con 0x7f81100b4a30 2023-11-08T08:20:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f810dffb700 1 -- 172.21.15.116:0/3736152313 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108003dd0 con 0x7f81100b4a30 2023-11-08T08:20:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f81154a0700 1 -- 172.21.15.116:0/3736152313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4a30 msgr2=0x7f81100b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.890+0000 7f81154a0700 1 --2- 172.21.15.116:0/3736152313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4a30 0x7f81100b4e90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f811006efd0 tx=0x7f810800b330 comp rx=0 tx=0).stop 2023-11-08T08:20:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.891+0000 7f81154a0700 1 -- 172.21.15.116:0/3736152313 shutdown_connections 2023-11-08T08:20:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.891+0000 7f81154a0700 1 --2- 172.21.15.116:0/3736152313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4a30 0x7f81100b4e90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.891+0000 7f81154a0700 1 --2- 172.21.15.116:0/3736152313 >> v2:172.21.15.129:3300/0 conn(0x7f81100b4660 0x7f8110071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.891+0000 7f81154a0700 1 -- 172.21.15.116:0/3736152313 >> 172.21.15.116:0/3736152313 conn(0x7f81100b2fc0 msgr2=0x7f8110071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.891+0000 7f81154a0700 1 -- 172.21.15.116:0/3736152313 shutdown_connections 2023-11-08T08:20:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.891+0000 7f81154a0700 1 -- 172.21.15.116:0/3736152313 wait complete. 2023-11-08T08:20:19.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.892+0000 7f81154a0700 1 Processor -- start 2023-11-08T08:20:19.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.892+0000 7f81154a0700 1 -- start start 2023-11-08T08:20:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 0x7f81100c3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81100b4a30 0x7f81101f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81100cff90 0x7f81100c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100ce110 con 0x7f81100b4660 2023-11-08T08:20:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100cd500 con 0x7f81100b4a30 2023-11-08T08:20:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100cd7e0 con 0x7f81100cff90 2023-11-08T08:20:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 0x7f81100c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 0x7f81100c3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56330/0 (socket says 172.21.15.116:56330) 2023-11-08T08:20:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 -- 172.21.15.116:0/3194718142 learned_addr learned my addr 172.21.15.116:0/3194718142 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810e7fc700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81100b4a30 0x7f81101f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810f7fe700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81100cff90 0x7f81100c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 -- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81100cff90 msgr2=0x7f81100c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81100cff90 0x7f81100c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 -- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81100b4a30 msgr2=0x7f81101f1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:19.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81100b4a30 0x7f81101f1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:19.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 -- 172.21.15.116:0/3194718142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108009b30 con 0x7f81100b4660 2023-11-08T08:20:19.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810f7fe700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81100cff90 0x7f81100c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:19.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f810effd700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 0x7f81100c3990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f810400c7f0 tx=0x7f810400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104004220 con 0x7f81100b4660 2023-11-08T08:20:19.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8104004380 con 0x7f81100b4660 2023-11-08T08:20:19.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811015f170 con 0x7f81100b4660 2023-11-08T08:20:19.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040036a0 con 0x7f81100b4660 2023-11-08T08:20:19.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.893+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81100cf790 con 0x7f81100b4660 2023-11-08T08:20:19.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.894+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811015f300 con 0x7f81100b4660 2023-11-08T08:20:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.895+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8104020070 con 0x7f81100b4660 2023-11-08T08:20:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.895+0000 7f80f77fe700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80fc0682f0 0x7f80fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.896+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8104085cd0 con 0x7f81100b4660 2023-11-08T08:20:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.896+0000 7f810e7fc700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80fc0682f0 0x7f80fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.896+0000 7f810e7fc700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80fc0682f0 0x7f80fc06a7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81100ada20 tx=0x7f81000090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:19.899+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8104052440 con 0x7f81100b4660 2023-11-08T08:20:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.121+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f811006b170 con 0x7f81100b4660 2023-11-08T08:20:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.122+0000 7f80f77fe700 1 -- 172.21.15.116:0/3194718142 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f810401e090 con 0x7f81100b4660 2023-11-08T08:20:20.123 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:20.124 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:20:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80fc0682f0 msgr2=0x7f80fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80fc0682f0 0x7f80fc06a7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81100ada20 tx=0x7f81000090e0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 msgr2=0x7f81100c3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 0x7f81100c3990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f810400c7f0 tx=0x7f810400cb00 comp rx=0 tx=0).stop 2023-11-08T08:20:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 shutdown_connections 2023-11-08T08:20:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81100cff90 0x7f81100c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81100b4660 0x7f81100c3990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80fc0682f0 0x7f80fc06a7a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 --2- 172.21.15.116:0/3194718142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81100b4a30 0x7f81101f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 >> 172.21.15.116:0/3194718142 conn(0x7f81100b2fc0 msgr2=0x7f811015e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 shutdown_connections 2023-11-08T08:20:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.124+0000 7f81154a0700 1 -- 172.21.15.116:0/3194718142 wait complete. 2023-11-08T08:20:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:20:20.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.255+0000 7f552267a700 1 Processor -- start 2023-11-08T08:20:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f552267a700 1 -- start start 2023-11-08T08:20:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f552267a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f551c070d90 0x7f551c0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f552267a700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c151a50 con 0x7f551c15e0b0 2023-11-08T08:20:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f552267a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c15b440 con 0x7f551c06a270 2023-11-08T08:20:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f552267a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c071ac0 con 0x7f551c070d90 2023-11-08T08:20:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f551b7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f551c070d90 0x7f551c0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.256+0000 7f5520c17700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f551c15e0b0 0x7f551c15e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58424/0 (socket says 172.21.15.116:58424) 2023-11-08T08:20:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f5520c17700 1 -- 172.21.15.116:0/2174971877 learned_addr learned my addr 172.21.15.116:0/2174971877 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551b7fe700 1 -- 172.21.15.116:0/2174971877 >> v1:172.21.15.129:6789/0 conn(0x7f551c06a270 legacy=0x7f551c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551b7fe700 1 -- 172.21.15.116:0/2174971877 >> v1:172.21.15.116:6789/0 conn(0x7f551c15e0b0 legacy=0x7f551c15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551affd700 1 -- 172.21.15.116:0/2174971877 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200130100 0 0) 0x7f551c151a50 con 0x7f551c15e0b0 2023-11-08T08:20:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551b7fe700 1 -- 172.21.15.116:0/2174971877 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c15adf0 con 0x7f551c070d90 2023-11-08T08:20:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551b7fe700 1 --2- 172.21.15.116:0/2174971877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c070d90 0x7f551c0711e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f550c009840 tx=0x7f550c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92596d761113f9c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551affd700 1 -- 172.21.15.116:0/2174971877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c011830 con 0x7f551c070d90 2023-11-08T08:20:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551affd700 1 -- 172.21.15.116:0/2174971877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c011990 con 0x7f551c070d90 2023-11-08T08:20:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.257+0000 7f551affd700 1 -- 172.21.15.116:0/2174971877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c011d20 con 0x7f551c070d90 2023-11-08T08:20:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 -- 172.21.15.116:0/2174971877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c070d90 msgr2=0x7f551c0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 --2- 172.21.15.116:0/2174971877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c070d90 0x7f551c0711e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f550c009840 tx=0x7f550c009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 -- 172.21.15.116:0/2174971877 shutdown_connections 2023-11-08T08:20:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 --2- 172.21.15.116:0/2174971877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c070d90 0x7f551c0711e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 -- 172.21.15.116:0/2174971877 >> 172.21.15.116:0/2174971877 conn(0x7f551c06c010 msgr2=0x7f551c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 -- 172.21.15.116:0/2174971877 shutdown_connections 2023-11-08T08:20:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.258+0000 7f552267a700 1 -- 172.21.15.116:0/2174971877 wait complete. 2023-11-08T08:20:20.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.259+0000 7f552267a700 1 Processor -- start 2023-11-08T08:20:20.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.259+0000 7f552267a700 1 -- start start 2023-11-08T08:20:20.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f552267a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c06a270 0x7f551c0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f552267a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f551c070d90 0x7f551c0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f552267a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 0x7f551c1f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f552267a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c0c8140 con 0x7f551c15e0b0 2023-11-08T08:20:20.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f552267a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c0c8420 con 0x7f551c06a270 2023-11-08T08:20:20.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f552267a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c0c8700 con 0x7f551c070d90 2023-11-08T08:20:20.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f5520c17700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 0x7f551c1f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f5520c17700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 0x7f551c1f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56338/0 (socket says 172.21.15.116:56338) 2023-11-08T08:20:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.260+0000 7f5520c17700 1 -- 172.21.15.116:0/1329088824 learned_addr learned my addr 172.21.15.116:0/1329088824 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5520c17700 1 -- 172.21.15.116:0/1329088824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f551c070d90 msgr2=0x7f551c0c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5520c17700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f551c070d90 0x7f551c0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5520c17700 1 -- 172.21.15.116:0/1329088824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c06a270 msgr2=0x7f551c0c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5520c17700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c06a270 0x7f551c0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5520c17700 1 -- 172.21.15.116:0/1329088824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c0094a0 con 0x7f551c15e0b0 2023-11-08T08:20:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5520c17700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 0x7f551c1f05e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f551400cba0 tx=0x7f551400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551400ab80 con 0x7f551c15e0b0 2023-11-08T08:20:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551400ad00 con 0x7f551c15e0b0 2023-11-08T08:20:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514007310 con 0x7f551c15e0b0 2023-11-08T08:20:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c15aac0 con 0x7f551c15e0b0 2023-11-08T08:20:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.261+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c15b0d0 con 0x7f551c15e0b0 2023-11-08T08:20:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.262+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c071720 con 0x7f551c15e0b0 2023-11-08T08:20:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.262+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5514010070 con 0x7f551c15e0b0 2023-11-08T08:20:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.263+0000 7f5518ff9700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55080682f0 0x7f550806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.263+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f55140856f0 con 0x7f551c15e0b0 2023-11-08T08:20:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.263+0000 7f551bfff700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55080682f0 0x7f550806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.263+0000 7f551bfff700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55080682f0 0x7f550806a7a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5510000f60 tx=0x7f5510007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.267+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5514051e60 con 0x7f551c15e0b0 2023-11-08T08:20:20.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.488+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f551c0743f0 con 0x7f551c15e0b0 2023-11-08T08:20:20.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.488+0000 7f5518ff9700 1 -- 172.21.15.116:0/1329088824 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f5514055480 con 0x7f551c15e0b0 2023-11-08T08:20:20.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:20.491 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:19:45.064539+0000","modified":"2023-11-08T08:19:48.075173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5265":{"gid":5265,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/2464372713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2464372713},{"type":"v1","addr":"172.21.15.116:6837","nonce":2464372713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-08T08:20:20.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.490+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55080682f0 msgr2=0x7f550806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.490+0000 7f552267a700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55080682f0 0x7f550806a7a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5510000f60 tx=0x7f5510007a40 comp rx=0 tx=0).stop 2023-11-08T08:20:20.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.490+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 msgr2=0x7f551c1f05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.490+0000 7f552267a700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 0x7f551c1f05e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f551400cba0 tx=0x7f551400cf60 comp rx=0 tx=0).stop 2023-11-08T08:20:20.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 shutdown_connections 2023-11-08T08:20:20.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f551c15e0b0 0x7f551c1f05e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f55080682f0 0x7f550806a7a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f551c06a270 0x7f551c0c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 --2- 172.21.15.116:0/1329088824 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f551c070d90 0x7f551c0c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 >> 172.21.15.116:0/1329088824 conn(0x7f551c06c010 msgr2=0x7f551c0cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 shutdown_connections 2023-11-08T08:20:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.491+0000 7f552267a700 1 -- 172.21.15.116:0/1329088824 wait complete. 2023-11-08T08:20:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2023-11-08T08:20:20.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.623+0000 7fd8fe0b7700 1 Processor -- start 2023-11-08T08:20:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.623+0000 7fd8fe0b7700 1 -- start start 2023-11-08T08:20:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8fe0b7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f8071f70 0x7fd8f80cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8fe0b7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd8f80b4a30 0x7fd8f80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8fe0b7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80cfb00 con 0x7fd8f80b4660 2023-11-08T08:20:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8fe0b7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f815aa20 con 0x7fd8f80b4a30 2023-11-08T08:20:20.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8fe0b7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80cf4d0 con 0x7fd8f8071f70 2023-11-08T08:20:20.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f7fff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd8f80b4a30 0x7fd8f80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f6ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f8071f70 0x7fd8f80cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f77fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd8f80b4660 0x7fd8f806b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:37126/0 (socket says 172.21.15.116:37126) 2023-11-08T08:20:20.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f77fe700 1 -- 172.21.15.116:0/1791103500 learned_addr learned my addr 172.21.15.116:0/1791103500 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:20.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f6ffd700 1 -- 172.21.15.116:0/1791103500 >> v2:172.21.15.129:3300/0 conn(0x7fd8f80b4a30 msgr2=0x7fd8f80b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:20.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f6ffd700 1 --2- 172.21.15.116:0/1791103500 >> v2:172.21.15.129:3300/0 conn(0x7fd8f80b4a30 0x7fd8f80b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f6ffd700 1 -- 172.21.15.116:0/1791103500 >> v1:172.21.15.129:6789/0 conn(0x7fd8f80b4660 legacy=0x7fd8f806b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f6ffd700 1 -- 172.21.15.116:0/1791103500 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8072550 con 0x7fd8f8071f70 2023-11-08T08:20:20.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f7fff700 1 --2- 172.21.15.116:0/1791103500 >> v2:172.21.15.129:3300/0 conn(0x7fd8f80b4a30 0x7fd8f80b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:20.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.624+0000 7fd8f6ffd700 1 --2- 172.21.15.116:0/1791103500 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f8071f70 0x7fd8f80cff90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009840 tx=0x7fd8ec009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=becba43fdbed0b66 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8f67fc700 1 -- 172.21.15.116:0/1791103500 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011830 con 0x7fd8f8071f70 2023-11-08T08:20:20.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8f67fc700 1 -- 172.21.15.116:0/1791103500 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec011990 con 0x7fd8f8071f70 2023-11-08T08:20:20.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8f67fc700 1 -- 172.21.15.116:0/1791103500 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011d20 con 0x7fd8f8071f70 2023-11-08T08:20:20.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/1791103500 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f8071f70 msgr2=0x7fd8f80cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/1791103500 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f8071f70 0x7fd8f80cff90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009840 tx=0x7fd8ec009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:20.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/1791103500 shutdown_connections 2023-11-08T08:20:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/1791103500 >> v2:172.21.15.129:3300/0 conn(0x7fd8f80b4a30 0x7fd8f80b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/1791103500 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f8071f70 0x7fd8f80cff90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/1791103500 >> 172.21.15.116:0/1791103500 conn(0x7fd8f80b2fc0 msgr2=0x7fd8f8071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/1791103500 shutdown_connections 2023-11-08T08:20:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.625+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/1791103500 wait complete. 2023-11-08T08:20:20.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.626+0000 7fd8fe0b7700 1 Processor -- start 2023-11-08T08:20:20.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.626+0000 7fd8fe0b7700 1 -- start start 2023-11-08T08:20:20.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8fe0b7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8071f70 0x7fd8f81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8fe0b7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f80b4660 0x7fd8f80c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8fe0b7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 0x7fd8f81f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8fe0b7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f815c2a0 con 0x7fd8f80b4a30 2023-11-08T08:20:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8fe0b7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80c57e0 con 0x7fd8f8071f70 2023-11-08T08:20:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8fe0b7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80c5500 con 0x7fd8f80b4660 2023-11-08T08:20:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8f7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 0x7fd8f81f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8f7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 0x7fd8f81f54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56342/0 (socket says 172.21.15.116:56342) 2023-11-08T08:20:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8f7fff700 1 -- 172.21.15.116:0/4107952486 learned_addr learned my addr 172.21.15.116:0/4107952486 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8f77fe700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8071f70 0x7fd8f81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.627+0000 7fd8f6ffd700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f80b4660 0x7fd8f80c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8f7fff700 1 -- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f80b4660 msgr2=0x7fd8f80c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8f7fff700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f80b4660 0x7fd8f80c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8f7fff700 1 -- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8071f70 msgr2=0x7fd8f81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8f7fff700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8071f70 0x7fd8f81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8f7fff700 1 -- 172.21.15.116:0/4107952486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f0009a70 con 0x7fd8f80b4a30 2023-11-08T08:20:20.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8f7fff700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 0x7fd8f81f54f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000efc0 tx=0x7fd8f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f000ca90 con 0x7fd8f80b4a30 2023-11-08T08:20:20.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f000f040 con 0x7fd8f80b4a30 2023-11-08T08:20:20.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec0094a0 con 0x7fd8f80b4a30 2023-11-08T08:20:20.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f000ce30 con 0x7fd8f80b4a30 2023-11-08T08:20:20.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.628+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f815f170 con 0x7fd8f80b4a30 2023-11-08T08:20:20.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.629+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f80cf790 con 0x7fd8f80b4a30 2023-11-08T08:20:20.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.629+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd8f0014480 con 0x7fd8f80b4a30 2023-11-08T08:20:20.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.630+0000 7fd8dffff700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8e40682f0 0x7fd8e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.630+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd8f00863e0 con 0x7fd8f80b4a30 2023-11-08T08:20:20.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.630+0000 7fd8f77fe700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8e40682f0 0x7fd8e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.630+0000 7fd8f77fe700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8e40682f0 0x7fd8e406a7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8005910 tx=0x7fd8e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.634+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8f0052be0 con 0x7fd8f80b4a30 2023-11-08T08:20:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.816+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8f815f300 con 0x7fd8f80b4a30 2023-11-08T08:20:20.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.818+0000 7fd8dffff700 1 -- 172.21.15.116:0/4107952486 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+17969 (secure 0 0 0) 0x7fd8f0009cd0 con 0x7fd8f80b4a30 2023-11-08T08:20:20.819 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:20.819 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:19:46.066442+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:19:43.321551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-11-08T08:19:44.023597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:20.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8e40682f0 msgr2=0x7fd8e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8e40682f0 0x7fd8e406a7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8005910 tx=0x7fd8e8009040 comp rx=0 tx=0).stop 2023-11-08T08:20:20.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 msgr2=0x7fd8f81f54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 0x7fd8f81f54f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000efc0 tx=0x7fd8f000c3a0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 shutdown_connections 2023-11-08T08:20:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8f80b4a30 0x7fd8f81f54f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd8e40682f0 0x7fd8e406a7a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8f8071f70 0x7fd8f81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 --2- 172.21.15.116:0/4107952486 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd8f80b4660 0x7fd8f80c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 >> 172.21.15.116:0/4107952486 conn(0x7fd8f80b2fc0 msgr2=0x7fd8f80ad2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:20.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 shutdown_connections 2023-11-08T08:20:20.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.820+0000 7fd8fe0b7700 1 -- 172.21.15.116:0/4107952486 wait complete. 2023-11-08T08:20:20.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-08T08:20:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.952+0000 7f964ce63700 1 Processor -- start 2023-11-08T08:20:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.952+0000 7f964ce63700 1 -- start start 2023-11-08T08:20:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.953+0000 7f964ce63700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f964806a270 0x7f964806a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.953+0000 7f964ce63700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964806abd0 con 0x7f96480718e0 2023-11-08T08:20:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.953+0000 7f964ce63700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9648160a50 con 0x7f964815c810 2023-11-08T08:20:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.953+0000 7f964ce63700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964815cbe0 con 0x7f964806a270 2023-11-08T08:20:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f9645d9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f96480718e0 0x7f964806c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58438/0 (socket says 172.21.15.116:58438) 2023-11-08T08:20:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f9645d9b700 1 -- 172.21.15.116:0/812984946 learned_addr learned my addr 172.21.15.116:0/812984946 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f9646d9d700 1 --2- 172.21.15.116:0/812984946 >> v2:172.21.15.129:3300/0 conn(0x7f964806a270 0x7f964806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740314864 0 0) 0x7f964806abd0 con 0x7f96480718e0 2023-11-08T08:20:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96300035b0 con 0x7f96480718e0 2023-11-08T08:20:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534360511 0 0) 0x7f96300035b0 con 0x7f96480718e0 2023-11-08T08:20:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 >> v2:172.21.15.129:3300/0 conn(0x7f964806a270 msgr2=0x7f964806a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 --2- 172.21.15.116:0/812984946 >> v2:172.21.15.129:3300/0 conn(0x7f964806a270 0x7f964806a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 >> v1:172.21.15.129:6789/0 conn(0x7f964815c810 legacy=0x7f9648160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648160d30 con 0x7f96480718e0 2023-11-08T08:20:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9638001670 con 0x7f96480718e0 2023-11-08T08:20:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 >> v1:172.21.15.116:6789/0 conn(0x7f96480718e0 legacy=0x7f964806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9630003a00 0x7f9630003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96300043f0 0x7f96300048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f963000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964815cbe0 con 0x7f96300050b0 2023-11-08T08:20:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963000c000 con 0x7f96300043f0 2023-11-08T08:20:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963000c1c0 con 0x7f9630003a00 2023-11-08T08:20:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f9646d9d700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f963000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f9645d9b700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96300043f0 0x7f96300048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.954+0000 7f964659c700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9630003a00 0x7f9630003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 -- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9630003a00 msgr2=0x7f9630003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9630003a00 0x7f9630003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 -- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96300043f0 msgr2=0x7f96300048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96300043f0 0x7f96300048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 -- 172.21.15.116:0/812984946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300035b0 con 0x7f96300050b0 2023-11-08T08:20:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 -- 172.21.15.116:0/812984946 reap_dead start 2023-11-08T08:20:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f9646d9d700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f963000ba10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9640009fc0 tx=0x7f9630005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd3eea3926147a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640003a20 con 0x7f96300050b0 2023-11-08T08:20:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964000f040 con 0x7f96300050b0 2023-11-08T08:20:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964559a700 1 -- 172.21.15.116:0/812984946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640003de0 con 0x7f96300050b0 2023-11-08T08:20:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 -- 172.21.15.116:0/812984946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 msgr2=0x7f963000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f963000ba10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9640009fc0 tx=0x7f9630005880 comp rx=0 tx=0).stop 2023-11-08T08:20:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 -- 172.21.15.116:0/812984946 shutdown_connections 2023-11-08T08:20:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 --2- 172.21.15.116:0/812984946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f963000ba10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 -- 172.21.15.116:0/812984946 >> 172.21.15.116:0/812984946 conn(0x7f964806c010 msgr2=0x7f96480cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 -- 172.21.15.116:0/812984946 shutdown_connections 2023-11-08T08:20:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.955+0000 7f964ce63700 1 -- 172.21.15.116:0/812984946 wait complete. 2023-11-08T08:20:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.957+0000 7f964ce63700 1 Processor -- start 2023-11-08T08:20:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.957+0000 7f964ce63700 1 -- start start 2023-11-08T08:20:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f96481f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f964815d490 0x7f96480b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f964815af80 0x7f964815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480b5940 con 0x7f96300050b0 2023-11-08T08:20:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964815b930 con 0x7f964815d490 2023-11-08T08:20:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480b2d50 con 0x7f964815af80 2023-11-08T08:20:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f96481f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f96481f1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56368/0 (socket says 172.21.15.116:56368) 2023-11-08T08:20:20.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 -- 172.21.15.116:0/3185936258 learned_addr learned my addr 172.21.15.116:0/3185936258 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:20.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f9646d9d700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f964815af80 0x7f964815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f9645d9b700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f964815d490 0x7f96480b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 -- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f964815af80 msgr2=0x7f964815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f964815af80 0x7f964815b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 -- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f964815d490 msgr2=0x7f96480b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:20.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f964815d490 0x7f96480b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:20.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 -- 172.21.15.116:0/3185936258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480ceb20 con 0x7f96300050b0 2023-11-08T08:20:20.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964659c700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f96481f1990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f963c00f7b0 tx=0x7f963c00fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c0052e0 con 0x7f96300050b0 2023-11-08T08:20:20.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963c005460 con 0x7f96300050b0 2023-11-08T08:20:20.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c0038c0 con 0x7f96300050b0 2023-11-08T08:20:20.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9648161010 con 0x7f96300050b0 2023-11-08T08:20:20.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.958+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96480ce740 con 0x7f96300050b0 2023-11-08T08:20:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.959+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96480743f0 con 0x7f96300050b0 2023-11-08T08:20:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.960+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f963c003a20 con 0x7f96300050b0 2023-11-08T08:20:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.960+0000 7f9636ffd700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96280682f0 0x7f962806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.960+0000 7f9645d9b700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96280682f0 0x7f962806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.960+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f963c085b20 con 0x7f96300050b0 2023-11-08T08:20:20.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.961+0000 7f9645d9b700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96280682f0 0x7f962806a7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9638001050 tx=0x7f9638001c30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:20.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:20.964+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f963c052380 con 0x7f96300050b0 2023-11-08T08:20:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.189+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f964815bfc0 con 0x7f96300050b0 2023-11-08T08:20:21.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-08T08:20:21.524+0000 7f50f4469700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-08T08:20:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.527+0000 7f9636ffd700 1 -- 172.21.15.116:0/3185936258 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f963c0559a0 con 0x7f96300050b0 2023-11-08T08:20:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96280682f0 msgr2=0x7f962806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96280682f0 0x7f962806a7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9638001050 tx=0x7f9638001c30 comp rx=0 tx=0).stop 2023-11-08T08:20:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 msgr2=0x7f96481f1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f96481f1990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f963c00f7b0 tx=0x7f963c00fac0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 shutdown_connections 2023-11-08T08:20:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f964815af80 0x7f964815b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96300050b0 0x7f96481f1990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f96280682f0 0x7f962806a7a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 --2- 172.21.15.116:0/3185936258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f964815d490 0x7f96480b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 >> 172.21.15.116:0/3185936258 conn(0x7f964806c010 msgr2=0x7f964806d470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:21.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 shutdown_connections 2023-11-08T08:20:21.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.530+0000 7f964ce63700 1 -- 172.21.15.116:0/3185936258 wait complete. 2023-11-08T08:20:21.534 INFO:tasks.ceph.mds.a.smithi116.stderr: -5544> 2023-11-08T08:19:48.576+0000 7fe0a79d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:20:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-08T08:20:21.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-08T08:20:21.564 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-08T08:20:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.667+0000 7fb3437ea700 1 Processor -- start 2023-11-08T08:20:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.667+0000 7fb3437ea700 1 -- start start 2023-11-08T08:20:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb3437ea700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb3437ea700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb33c0c9a50 0x7fb33c0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb3437ea700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c06a310 con 0x7fb33c159710 2023-11-08T08:20:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb3437ea700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c0707c0 con 0x7fb33c0c9a50 2023-11-08T08:20:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb3437ea700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0ca410 con 0x7fb33c070370 2023-11-08T08:20:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341586700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb33c159710 0x7fb33c0cdff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58460/0 (socket says 172.21.15.116:58460) 2023-11-08T08:20:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341586700 1 -- 172.21.15.116:0/1843445315 learned_addr learned my addr 172.21.15.116:0/1843445315 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341d87700 1 --2- 172.21.15.116:0/1843445315 >> v2:172.21.15.116:3300/0 conn(0x7fb33c0c9a50 0x7fb33c0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb333fff700 1 -- 172.21.15.116:0/1843445315 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288928194 0 0) 0x7fb33c06a310 con 0x7fb33c159710 2023-11-08T08:20:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb333fff700 1 -- 172.21.15.116:0/1843445315 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3280035b0 con 0x7fb33c159710 2023-11-08T08:20:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb340d85700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341d87700 1 -- 172.21.15.116:0/1843445315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 msgr2=0x7fb33c0cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341d87700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c0cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341d87700 1 -- 172.21.15.116:0/1843445315 >> v1:172.21.15.116:6789/0 conn(0x7fb33c159710 legacy=0x7fb33c0cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb341d87700 1 -- 172.21.15.116:0/1843445315 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0ce7e0 con 0x7fb33c0c9a50 2023-11-08T08:20:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.668+0000 7fb340d85700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb341d87700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c0c9a50 0x7fb33c0c9ed0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d90 tx=0x7fb33400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9f1d7ec61b828 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb333fff700 1 -- 172.21.15.116:0/1843445315 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400e070 con 0x7fb33c0c9a50 2023-11-08T08:20:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb333fff700 1 -- 172.21.15.116:0/1843445315 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33400bb20 con 0x7fb33c0c9a50 2023-11-08T08:20:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb333fff700 1 -- 172.21.15.116:0/1843445315 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400be90 con 0x7fb33c0c9a50 2023-11-08T08:20:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 -- 172.21.15.116:0/1843445315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c0c9a50 msgr2=0x7fb33c0c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c0c9a50 0x7fb33c0c9ed0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d90 tx=0x7fb33400b330 comp rx=0 tx=0).stop 2023-11-08T08:20:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 -- 172.21.15.116:0/1843445315 shutdown_connections 2023-11-08T08:20:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c0c9a50 0x7fb33c0c9ed0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 --2- 172.21.15.116:0/1843445315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 -- 172.21.15.116:0/1843445315 >> 172.21.15.116:0/1843445315 conn(0x7fb33c0cf400 msgr2=0x7fb33c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 -- 172.21.15.116:0/1843445315 shutdown_connections 2023-11-08T08:20:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.669+0000 7fb3437ea700 1 -- 172.21.15.116:0/1843445315 wait complete. 2023-11-08T08:20:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.671+0000 7fb3437ea700 1 Processor -- start 2023-11-08T08:20:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.671+0000 7fb3437ea700 1 -- start start 2023-11-08T08:20:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb3437ea700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb3437ea700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33c0c9a50 0x7fb33c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb3437ea700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 0x7fb33c155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb3437ea700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c155470 con 0x7fb33c159710 2023-11-08T08:20:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb3437ea700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c154300 con 0x7fb33c0c9a50 2023-11-08T08:20:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb3437ea700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1544e0 con 0x7fb33c070370 2023-11-08T08:20:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 0x7fb33c155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 0x7fb33c155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56406/0 (socket says 172.21.15.116:56406) 2023-11-08T08:20:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 -- 172.21.15.116:0/3555948578 learned_addr learned my addr 172.21.15.116:0/3555948578 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb340d85700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33c0c9a50 0x7fb33c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341586700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 -- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 msgr2=0x7fb33c154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:21.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 -- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33c0c9a50 msgr2=0x7fb33c153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:21.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33c0c9a50 0x7fb33c153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:21.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 -- 172.21.15.116:0/3555948578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334009a70 con 0x7fb33c159710 2023-11-08T08:20:21.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb340d85700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33c0c9a50 0x7fb33c153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:21.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341586700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb341d87700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 0x7fb33c155920 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb33400b680 tx=0x7fb334004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.672+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3340098f0 con 0x7fb33c159710 2023-11-08T08:20:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.673+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334020070 con 0x7fb33c159710 2023-11-08T08:20:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.673+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c06a890 con 0x7fb33c159710 2023-11-08T08:20:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.673+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3340276a0 con 0x7fb33c159710 2023-11-08T08:20:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.673+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c06a5b0 con 0x7fb33c159710 2023-11-08T08:20:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.674+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c0ceac0 con 0x7fb33c159710 2023-11-08T08:20:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.674+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb33402e070 con 0x7fb33c159710 2023-11-08T08:20:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.674+0000 7fb331ffb700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3200682f0 0x7fb32006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.674+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb334022070 con 0x7fb33c159710 2023-11-08T08:20:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.674+0000 7fb341586700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3200682f0 0x7fb32006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.675+0000 7fb341586700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3200682f0 0x7fb32006a7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb32c005910 tx=0x7fb32c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.679+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb334054240 con 0x7fb33c159710 2023-11-08T08:20:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:21.900+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb33c0743f0 con 0x7fb33c159710 2023-11-08T08:20:22.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.535+0000 7fb331ffb700 1 -- 172.21.15.116:0/3555948578 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb33405be60 con 0x7fb33c159710 2023-11-08T08:20:22.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3200682f0 msgr2=0x7fb32006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:22.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3200682f0 0x7fb32006a7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb32c005910 tx=0x7fb32c009040 comp rx=0 tx=0).stop 2023-11-08T08:20:22.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 msgr2=0x7fb33c155920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:22.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 0x7fb33c155920 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb33400b680 tx=0x7fb334004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 shutdown_connections 2023-11-08T08:20:22.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb33c159710 0x7fb33c155920 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb33c070370 0x7fb33c154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3200682f0 0x7fb32006a7a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 --2- 172.21.15.116:0/3555948578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33c0c9a50 0x7fb33c153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 >> 172.21.15.116:0/3555948578 conn(0x7fb33c0cf400 msgr2=0x7fb33c0afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:22.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 shutdown_connections 2023-11-08T08:20:22.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.538+0000 7fb3437ea700 1 -- 172.21.15.116:0/3555948578 wait complete. 2023-11-08T08:20:22.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-08T08:20:22.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.725+0000 7f57e90e4700 1 Processor -- start 2023-11-08T08:20:22.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.725+0000 7f57e90e4700 1 -- start start 2023-11-08T08:20:22.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e90e4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57e415e070 0x7f57e406b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:22.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e90e4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f57e40ceea0 0x7f57e406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:22.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e90e4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40cea10 con 0x7f57e415a150 2023-11-08T08:20:22.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e90e4700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e415ae80 con 0x7f57e415e070 2023-11-08T08:20:22.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e90e4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e40cbd10 con 0x7f57e40ceea0 2023-11-08T08:20:22.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e2d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57e415e070 0x7f57e406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:22.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e359e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f57e415a150 0x7f57e415a5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39824/0 (socket says 172.21.15.116:39824) 2023-11-08T08:20:22.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e359e700 1 -- 172.21.15.116:0/3173901122 learned_addr learned my addr 172.21.15.116:0/3173901122 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:22.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e259c700 1 --2- 172.21.15.116:0/3173901122 >> v2:172.21.15.129:3300/0 conn(0x7f57e40ceea0 0x7f57e406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:22.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e2d9d700 1 -- 172.21.15.116:0/3173901122 >> v2:172.21.15.129:3300/0 conn(0x7f57e40ceea0 msgr2=0x7f57e406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:22.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e2d9d700 1 --2- 172.21.15.116:0/3173901122 >> v2:172.21.15.129:3300/0 conn(0x7f57e40ceea0 0x7f57e406b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e2d9d700 1 -- 172.21.15.116:0/3173901122 >> v1:172.21.15.116:6789/0 conn(0x7f57e415a150 legacy=0x7f57e415a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:22.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e2d9d700 1 -- 172.21.15.116:0/3173901122 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40ce0e0 con 0x7f57e415e070 2023-11-08T08:20:22.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e1d9b700 1 -- 172.21.15.116:0/3173901122 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280894614 0 0) 0x7f57e40cea10 con 0x7f57e415a150 2023-11-08T08:20:22.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e2d9d700 1 --2- 172.21.15.116:0/3173901122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e415e070 0x7f57e406b380 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f57e40ca790 tx=0x7f57d4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f79dcba152dafc3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:22.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e1d9b700 1 -- 172.21.15.116:0/3173901122 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4011830 con 0x7f57e415e070 2023-11-08T08:20:22.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e1d9b700 1 -- 172.21.15.116:0/3173901122 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d4011990 con 0x7f57e415e070 2023-11-08T08:20:22.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.726+0000 7f57e1d9b700 1 -- 172.21.15.116:0/3173901122 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4011d20 con 0x7f57e415e070 2023-11-08T08:20:22.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 -- 172.21.15.116:0/3173901122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e415e070 msgr2=0x7f57e406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 --2- 172.21.15.116:0/3173901122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e415e070 0x7f57e406b380 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f57e40ca790 tx=0x7f57d4009b20 comp rx=0 tx=0).stop 2023-11-08T08:20:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 -- 172.21.15.116:0/3173901122 shutdown_connections 2023-11-08T08:20:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 --2- 172.21.15.116:0/3173901122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e415e070 0x7f57e406b380 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 --2- 172.21.15.116:0/3173901122 >> v2:172.21.15.129:3300/0 conn(0x7f57e40ceea0 0x7f57e406b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 -- 172.21.15.116:0/3173901122 >> 172.21.15.116:0/3173901122 conn(0x7f57e40cfc30 msgr2=0x7f57e40d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:22.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 -- 172.21.15.116:0/3173901122 shutdown_connections 2023-11-08T08:20:22.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.727+0000 7f57e90e4700 1 -- 172.21.15.116:0/3173901122 wait complete. 2023-11-08T08:20:22.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.728+0000 7f57e90e4700 1 Processor -- start 2023-11-08T08:20:22.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.728+0000 7f57e90e4700 1 -- start start 2023-11-08T08:20:22.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e90e4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 0x7f57e40b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:22.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e90e4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e415a150 0x7f57e414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:22.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e90e4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e415e070 0x7f57e414c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:22.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e90e4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415cc90 con 0x7f57e40ceea0 2023-11-08T08:20:22.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e90e4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e414b7b0 con 0x7f57e415a150 2023-11-08T08:20:22.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e90e4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415c6d0 con 0x7f57e415e070 2023-11-08T08:20:22.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 0x7f57e40b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:22.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 0x7f57e40b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39380/0 (socket says 172.21.15.116:39380) 2023-11-08T08:20:22.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 -- 172.21.15.116:0/1368315374 learned_addr learned my addr 172.21.15.116:0/1368315374 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:22.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e259c700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e415a150 0x7f57e414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:22.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e359e700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e415e070 0x7f57e414c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:22.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 -- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e415e070 msgr2=0x7f57e414c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:22.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e415e070 0x7f57e414c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 -- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e415a150 msgr2=0x7f57e414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:22.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e415a150 0x7f57e414bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:22.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.729+0000 7f57e2d9d700 1 -- 172.21.15.116:0/1368315374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d40094a0 con 0x7f57e40ceea0 2023-11-08T08:20:22.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57e359e700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e415e070 0x7f57e414c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:22.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57e2d9d700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 0x7f57e40b3770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f57d4014040 tx=0x7f57d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:22.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d40038a0 con 0x7f57e40ceea0 2023-11-08T08:20:22.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d401e070 con 0x7f57e40ceea0 2023-11-08T08:20:22.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e40ce6a0 con 0x7f57e40ceea0 2023-11-08T08:20:22.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4003c60 con 0x7f57e40ceea0 2023-11-08T08:20:22.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.730+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e40ce3c0 con 0x7f57e40ceea0 2023-11-08T08:20:22.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.731+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e4150080 con 0x7f57e40ceea0 2023-11-08T08:20:22.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.731+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57d400f050 con 0x7f57e40ceea0 2023-11-08T08:20:22.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.731+0000 7f57cb7fe700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57d00682f0 0x7f57d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:22.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.732+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f57d4086f50 con 0x7f57e40ceea0 2023-11-08T08:20:22.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.732+0000 7f57e259c700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57d00682f0 0x7f57d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:22.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.732+0000 7f57e259c700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57d00682f0 0x7f57d006a7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f57e41f25b0 tx=0x7f57d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:22.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.736+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57d40536f0 con 0x7f57e40ceea0 2023-11-08T08:20:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:22.916+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57e414b610 con 0x7f57e40ceea0 2023-11-08T08:20:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.591+0000 7f57cb7fe700 1 -- 172.21.15.116:0/1368315374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) v1 ==== 181+0+0 (secure 0 0 0) 0x7f57e414b610 con 0x7f57e40ceea0 2023-11-08T08:20:23.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57d00682f0 msgr2=0x7f57d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:23.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57d00682f0 0x7f57d006a7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f57e41f25b0 tx=0x7f57d8008040 comp rx=0 tx=0).stop 2023-11-08T08:20:23.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 msgr2=0x7f57e40b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 0x7f57e40b3770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f57d4014040 tx=0x7f57d4004750 comp rx=0 tx=0).stop 2023-11-08T08:20:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 shutdown_connections 2023-11-08T08:20:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f57e415e070 0x7f57e414c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e40ceea0 0x7f57e40b3770 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f57d00682f0 0x7f57d006a7a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 --2- 172.21.15.116:0/1368315374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57e415a150 0x7f57e414bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 >> 172.21.15.116:0/1368315374 conn(0x7f57e40cfc30 msgr2=0x7f57e40d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 shutdown_connections 2023-11-08T08:20:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.594+0000 7f57e90e4700 1 -- 172.21.15.116:0/1368315374 wait complete. 2023-11-08T08:20:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2023-11-08T08:20:23.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-08T08:20:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.741+0000 7f937509e700 1 Processor -- start 2023-11-08T08:20:23.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.741+0000 7f937509e700 1 -- start start 2023-11-08T08:20:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f937509e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f937015cce0 0x7f937006b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f937509e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 0x7f9370160790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f937509e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93700cdc10 con 0x7f937015dc40 2023-11-08T08:20:23.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f937509e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937015d0b0 con 0x7f937015cce0 2023-11-08T08:20:23.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f937509e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93700cfa10 con 0x7f9370071320 2023-11-08T08:20:23.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936e59c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f937015dc40 0x7f9370070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39838/0 (socket says 172.21.15.116:39838) 2023-11-08T08:20:23.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936e59c700 1 -- 172.21.15.116:0/4093569819 learned_addr learned my addr 172.21.15.116:0/4093569819 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:23.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936ed9d700 1 --2- 172.21.15.116:0/4093569819 >> v2:172.21.15.129:3300/0 conn(0x7f937015cce0 0x7f937006b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165165704 0 0) 0x7f93700cdc10 con 0x7f937015dc40 2023-11-08T08:20:23.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936f59e700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 0x7f9370160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93580035b0 con 0x7f937015dc40 2023-11-08T08:20:23.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4228828909 0 0) 0x7f93580035b0 con 0x7f937015dc40 2023-11-08T08:20:23.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 msgr2=0x7f9370160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:23.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 0x7f9370160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 >> v2:172.21.15.129:3300/0 conn(0x7f937015cce0 msgr2=0x7f937006b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:23.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 --2- 172.21.15.116:0/4093569819 >> v2:172.21.15.129:3300/0 conn(0x7f937015cce0 0x7f937006b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93700cf730 con 0x7f937015dc40 2023-11-08T08:20:23.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.742+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9360001670 con 0x7f937015dc40 2023-11-08T08:20:23.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 >> v1:172.21.15.116:6789/0 conn(0x7f937015dc40 legacy=0x7f9370070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:23.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 0x7f9358003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f9358004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9358004d90 0x7f9358005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937015d0b0 con 0x7f9358003f80 2023-11-08T08:20:23.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93700cfa10 con 0x7f9358004d90 2023-11-08T08:20:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935800ba40 con 0x7f9370071320 2023-11-08T08:20:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f9358004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936f59e700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9358004d90 0x7f9358005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936ed9d700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 0x7f9358003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 -- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 msgr2=0x7f9358003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9370071320 0x7f9358003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 -- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9358004d90 msgr2=0x7f9358005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9358004d90 0x7f9358005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 -- 172.21.15.116:0/4093569819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93580035b0 con 0x7f9358003f80 2023-11-08T08:20:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936e59c700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f9358004850 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f936000c320 tx=0x7f9358005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec02814981177fd server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936000ce00 con 0x7f9358003f80 2023-11-08T08:20:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9360005960 con 0x7f9358003f80 2023-11-08T08:20:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.743+0000 7f936dd9b700 1 -- 172.21.15.116:0/4093569819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360005d10 con 0x7f9358003f80 2023-11-08T08:20:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f937509e700 1 -- 172.21.15.116:0/4093569819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 msgr2=0x7f9358004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f937509e700 1 --2- 172.21.15.116:0/4093569819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f9358004850 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f936000c320 tx=0x7f9358005880 comp rx=0 tx=0).stop 2023-11-08T08:20:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f936f59e700 1 -- 172.21.15.116:0/4093569819 reap_dead start 2023-11-08T08:20:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f937509e700 1 -- 172.21.15.116:0/4093569819 shutdown_connections 2023-11-08T08:20:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f937509e700 1 -- 172.21.15.116:0/4093569819 >> 172.21.15.116:0/4093569819 conn(0x7f93700cfc30 msgr2=0x7f93700d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f937509e700 1 -- 172.21.15.116:0/4093569819 shutdown_connections 2023-11-08T08:20:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.744+0000 7f937509e700 1 -- 172.21.15.116:0/4093569819 wait complete. 2023-11-08T08:20:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.746+0000 7f937509e700 1 Processor -- start 2023-11-08T08:20:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.746+0000 7f937509e700 1 -- start start 2023-11-08T08:20:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f93701571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93700b5490 0x7f93700cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93700b42e0 0x7f93700cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93700b35b0 con 0x7f9358003f80 2023-11-08T08:20:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93700b5920 con 0x7f93700b5490 2023-11-08T08:20:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93700b3800 con 0x7f93700b42e0 2023-11-08T08:20:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f93701571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f93701571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39396/0 (socket says 172.21.15.116:39396) 2023-11-08T08:20:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 -- 172.21.15.116:0/1515482455 learned_addr learned my addr 172.21.15.116:0/1515482455 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 -- 172.21.15.116:0/1515482455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93700b42e0 msgr2=0x7f93700cc770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:20:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93700b42e0 0x7f93700cc770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 -- 172.21.15.116:0/1515482455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93700b5490 msgr2=0x7f93700cc230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:20:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93700b5490 0x7f93700cc230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 -- 172.21.15.116:0/1515482455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93700cf3e0 con 0x7f9358003f80 2023-11-08T08:20:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f936ed9d700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f93701571d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9364011a20 tx=0x7f9364011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936400f940 con 0x7f9358003f80 2023-11-08T08:20:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936400fac0 con 0x7f9358003f80 2023-11-08T08:20:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936400fe70 con 0x7f9358003f80 2023-11-08T08:20:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93700cd8a0 con 0x7f9358003f80 2023-11-08T08:20:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.747+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937015e090 con 0x7f9358003f80 2023-11-08T08:20:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.748+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93700b4130 con 0x7f9358003f80 2023-11-08T08:20:23.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.749+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9364014440 con 0x7f9358003f80 2023-11-08T08:20:23.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.749+0000 7f93577fe700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f935c0682f0 0x7f935c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:23.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.749+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5710+0+0 (secure 0 0 0) 0x7f9364086370 con 0x7f9358003f80 2023-11-08T08:20:23.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.749+0000 7f936e59c700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f935c0682f0 0x7f935c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:23.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.749+0000 7f936e59c700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f935c0682f0 0x7f935c06a7a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f93700b5ca0 tx=0x7f936000a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:23.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.753+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9364052e40 con 0x7f9358003f80 2023-11-08T08:20:23.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:23.936+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f93700743f0 con 0x7f9358003f80 2023-11-08T08:20:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.593+0000 7f93577fe700 1 -- 172.21.15.116:0/1515482455 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9364056460 con 0x7f9358003f80 2023-11-08T08:20:24.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.596+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f935c0682f0 msgr2=0x7f935c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.596+0000 7f937509e700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f935c0682f0 0x7f935c06a7a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f93700b5ca0 tx=0x7f936000a000 comp rx=0 tx=0).stop 2023-11-08T08:20:24.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.596+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 msgr2=0x7f93701571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.596+0000 7f937509e700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f93701571d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9364011a20 tx=0x7f9364011de0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 shutdown_connections 2023-11-08T08:20:24.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93700b42e0 0x7f93700cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9358003f80 0x7f93701571d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f935c0682f0 0x7f935c06a7a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 --2- 172.21.15.116:0/1515482455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93700b5490 0x7f93700cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 >> 172.21.15.116:0/1515482455 conn(0x7f93700cfc30 msgr2=0x7f93700aea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:24.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 shutdown_connections 2023-11-08T08:20:24.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.597+0000 7f937509e700 1 -- 172.21.15.116:0/1515482455 wait complete. 2023-11-08T08:20:24.601 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2023-11-08T08:20:24.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-08T08:20:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.740+0000 7f9162edb700 1 Processor -- start 2023-11-08T08:20:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.740+0000 7f9162edb700 1 -- start start 2023-11-08T08:20:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9162edb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f915c070370 0x7f915c0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9162edb700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f915c0c9a50 0x7f915c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9162edb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c0ca3f0 con 0x7f915c159710 2023-11-08T08:20:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9162edb700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c0707f0 con 0x7f915c070370 2023-11-08T08:20:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9162edb700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c06a310 con 0x7f915c0c9a50 2023-11-08T08:20:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f915c070370 0x7f915c0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f915c070370 0x7f915c0cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39412/0 (socket says 172.21.15.116:39412) 2023-11-08T08:20:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 -- 172.21.15.116:0/295379893 learned_addr learned my addr 172.21.15.116:0/295379893 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9161478700 1 --2- 172.21.15.116:0/295379893 >> v2:172.21.15.129:3300/0 conn(0x7f915c0c9a50 0x7f915c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 -- 172.21.15.116:0/295379893 >> v2:172.21.15.129:3300/0 conn(0x7f915c0c9a50 msgr2=0x7f915c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 --2- 172.21.15.116:0/295379893 >> v2:172.21.15.129:3300/0 conn(0x7f915c0c9a50 0x7f915c0c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 -- 172.21.15.116:0/295379893 >> v1:172.21.15.129:6789/0 conn(0x7f915c159710 legacy=0x7f915c0cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f915bfff700 1 -- 172.21.15.116:0/295379893 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c0ce7e0 con 0x7f915c070370 2023-11-08T08:20:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.741+0000 7f9161478700 1 --2- 172.21.15.116:0/295379893 >> v2:172.21.15.129:3300/0 conn(0x7f915c0c9a50 0x7f915c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f915bfff700 1 --2- 172.21.15.116:0/295379893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c0cee40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f914800bb90 tx=0x7f914800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee523a2b778cd9ab server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f915b7fe700 1 -- 172.21.15.116:0/295379893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914800dd10 con 0x7f915c070370 2023-11-08T08:20:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f915b7fe700 1 -- 172.21.15.116:0/295379893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914800de70 con 0x7f915c070370 2023-11-08T08:20:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f915b7fe700 1 -- 172.21.15.116:0/295379893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148004230 con 0x7f915c070370 2023-11-08T08:20:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 -- 172.21.15.116:0/295379893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 msgr2=0x7f915c0cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 --2- 172.21.15.116:0/295379893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c0cee40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f914800bb90 tx=0x7f914800bea0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 -- 172.21.15.116:0/295379893 shutdown_connections 2023-11-08T08:20:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 --2- 172.21.15.116:0/295379893 >> v2:172.21.15.129:3300/0 conn(0x7f915c0c9a50 0x7f915c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 --2- 172.21.15.116:0/295379893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c0cee40 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 -- 172.21.15.116:0/295379893 >> 172.21.15.116:0/295379893 conn(0x7f915c0cf400 msgr2=0x7f915c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 -- 172.21.15.116:0/295379893 shutdown_connections 2023-11-08T08:20:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.742+0000 7f9162edb700 1 -- 172.21.15.116:0/295379893 wait complete. 2023-11-08T08:20:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.743+0000 7f9162edb700 1 Processor -- start 2023-11-08T08:20:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 -- start start 2023-11-08T08:20:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c14ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f915c0c9a50 0x7f915c1e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f915c159710 0x7f915c0b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0ad1c0 con 0x7f915c070370 2023-11-08T08:20:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0ad4a0 con 0x7f915c159710 2023-11-08T08:20:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9162edb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c1e3870 con 0x7f915c0c9a50 2023-11-08T08:20:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.744+0000 7f9160c77700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c14ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c14ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39420/0 (socket says 172.21.15.116:39420) 2023-11-08T08:20:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 -- 172.21.15.116:0/3145191929 learned_addr learned my addr 172.21.15.116:0/3145191929 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9161478700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f915c159710 0x7f915c0b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f915bfff700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f915c0c9a50 0x7f915c1e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 -- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f915c0c9a50 msgr2=0x7f915c1e4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f915c0c9a50 0x7f915c1e4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 -- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f915c159710 msgr2=0x7f915c0b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f915c159710 0x7f915c0b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 -- 172.21.15.116:0/3145191929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914800b7f0 con 0x7f915c070370 2023-11-08T08:20:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9161478700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f915c159710 0x7f915c0b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9160c77700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c14ec20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f915000c4d0 tx=0x7f915000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150011790 con 0x7f915c070370 2023-11-08T08:20:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91500118f0 con 0x7f915c070370 2023-11-08T08:20:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c06a890 con 0x7f915c070370 2023-11-08T08:20:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150011ca0 con 0x7f915c070370 2023-11-08T08:20:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.745+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c06a5b0 con 0x7f915c070370 2023-11-08T08:20:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.746+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c0ceac0 con 0x7f915c070370 2023-11-08T08:20:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.746+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9150010050 con 0x7f915c070370 2023-11-08T08:20:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.747+0000 7f91597fa700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f91400682f0 0x7f914006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.747+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f9150085680 con 0x7f915c070370 2023-11-08T08:20:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.747+0000 7f915bfff700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f91400682f0 0x7f914006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.747+0000 7f915bfff700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f91400682f0 0x7f914006a7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f915c0afdb0 tx=0x7f9148013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.751+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91500521e0 con 0x7f915c070370 2023-11-08T08:20:24.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.931+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f915c0743f0 con 0x7f915c070370 2023-11-08T08:20:24.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.935+0000 7f91597fa700 1 -- 172.21.15.116:0/3145191929 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9150085030 con 0x7f915c070370 2023-11-08T08:20:24.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f91400682f0 msgr2=0x7f914006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f91400682f0 0x7f914006a7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f915c0afdb0 tx=0x7f9148013000 comp rx=0 tx=0).stop 2023-11-08T08:20:24.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 msgr2=0x7f915c14ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:24.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c14ec20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f915000c4d0 tx=0x7f915000c890 comp rx=0 tx=0).stop 2023-11-08T08:20:24.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 shutdown_connections 2023-11-08T08:20:24.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f91400682f0 0x7f914006a7a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f915c159710 0x7f915c0b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c070370 0x7f915c14ec20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 --2- 172.21.15.116:0/3145191929 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f915c0c9a50 0x7f915c1e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 >> 172.21.15.116:0/3145191929 conn(0x7f915c0cf400 msgr2=0x7f915c071460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 shutdown_connections 2023-11-08T08:20:24.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:24.937+0000 7f9162edb700 1 -- 172.21.15.116:0/3145191929 wait complete. 2023-11-08T08:20:24.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2023-11-08T08:20:25.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.075+0000 7fae00563700 1 Processor -- start 2023-11-08T08:20:25.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.075+0000 7fae00563700 1 -- start start 2023-11-08T08:20:25.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fae00563700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fadf806a270 0x7fadf806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fae00563700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 0x7fadf8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fae00563700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf806ab80 con 0x7fadf815cc80 2023-11-08T08:20:25.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fae00563700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf80b2490 con 0x7fadf806a270 2023-11-08T08:20:25.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fae00563700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80b1810 con 0x7fadf806e570 2023-11-08T08:20:25.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fadf806a270 0x7fadf806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fadf806a270 0x7fadf806a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39424/0 (socket says 172.21.15.116:39424) 2023-11-08T08:20:25.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 -- 172.21.15.116:0/1880330596 learned_addr learned my addr 172.21.15.116:0/1880330596 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:25.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 -- 172.21.15.116:0/1880330596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 msgr2=0x7fadf8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:25.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 --2- 172.21.15.116:0/1880330596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 0x7fadf8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 -- 172.21.15.116:0/1880330596 >> v1:172.21.15.129:6789/0 conn(0x7fadf815cc80 legacy=0x7fadf806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:25.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 -- 172.21.15.116:0/1880330596 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80b1af0 con 0x7fadf806a270 2023-11-08T08:20:25.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfe2ff700 1 --2- 172.21.15.116:0/1880330596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf806a640 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fadf40098f0 tx=0x7fadf4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6643cd7e41c29190 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:25.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfd2fd700 1 -- 172.21.15.116:0/1880330596 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4004230 con 0x7fadf806a270 2023-11-08T08:20:25.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfd2fd700 1 -- 172.21.15.116:0/1880330596 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf4004390 con 0x7fadf806a270 2023-11-08T08:20:25.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.076+0000 7fadfd2fd700 1 -- 172.21.15.116:0/1880330596 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40036a0 con 0x7fadf806a270 2023-11-08T08:20:25.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 -- 172.21.15.116:0/1880330596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 msgr2=0x7fadf806a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 --2- 172.21.15.116:0/1880330596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf806a640 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fadf40098f0 tx=0x7fadf4009c00 comp rx=0 tx=0).stop 2023-11-08T08:20:25.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 -- 172.21.15.116:0/1880330596 shutdown_connections 2023-11-08T08:20:25.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 --2- 172.21.15.116:0/1880330596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf806a640 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 --2- 172.21.15.116:0/1880330596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 0x7fadf8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 -- 172.21.15.116:0/1880330596 >> 172.21.15.116:0/1880330596 conn(0x7fadf806c010 msgr2=0x7fadf80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 -- 172.21.15.116:0/1880330596 shutdown_connections 2023-11-08T08:20:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.077+0000 7fae00563700 1 -- 172.21.15.116:0/1880330596 wait complete. 2023-11-08T08:20:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.078+0000 7fae00563700 1 Processor -- start 2023-11-08T08:20:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.078+0000 7fae00563700 1 -- start start 2023-11-08T08:20:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf8153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 0x7fadf81f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadf815cc80 0x7fadf81f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81f5a90 con 0x7fadf806a270 2023-11-08T08:20:25.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81f2a30 con 0x7fadf815cc80 2023-11-08T08:20:25.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81f2d10 con 0x7fadf806e570 2023-11-08T08:20:25.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf8153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf8153230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39440/0 (socket says 172.21.15.116:39440) 2023-11-08T08:20:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 -- 172.21.15.116:0/526744127 learned_addr learned my addr 172.21.15.116:0/526744127 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 -- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 msgr2=0x7fadf81f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 0x7fadf81f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 -- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadf815cc80 msgr2=0x7fadf81f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfeb00700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadf815cc80 0x7fadf81f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadf815cc80 0x7fadf81f5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 -- 172.21.15.116:0/526744127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4009550 con 0x7fadf806a270 2023-11-08T08:20:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadfe2ff700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf8153230 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009130 tx=0x7fadf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400b660 con 0x7fadf806a270 2023-11-08T08:20:25.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf400b7e0 con 0x7fadf806a270 2023-11-08T08:20:25.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf80b20b0 con 0x7fadf806a270 2023-11-08T08:20:25.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400bb90 con 0x7fadf806a270 2023-11-08T08:20:25.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.079+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf80b1dd0 con 0x7fadf806a270 2023-11-08T08:20:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.080+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf815daa0 con 0x7fadf806a270 2023-11-08T08:20:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.081+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fadf4017020 con 0x7fadf806a270 2023-11-08T08:20:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.081+0000 7fadeeffd700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faddc0682f0 0x7faddc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.081+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fadf40870e0 con 0x7fadf806a270 2023-11-08T08:20:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.081+0000 7fadfdafe700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faddc0682f0 0x7faddc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.082+0000 7fadfdafe700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faddc0682f0 0x7faddc06a7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fadf8153770 tx=0x7fade8007a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.086+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadf4053c40 con 0x7fadf806a270 2023-11-08T08:20:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.340+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7fadf81f5d70 con 0x7fadf806a270 2023-11-08T08:20:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.596+0000 7fadeeffd700 1 -- 172.21.15.116:0/526744127 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v109) v1 ==== 142+0+0 (secure 0 0 0) 0x7fadf400bcf0 con 0x7fadf806a270 2023-11-08T08:20:25.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faddc0682f0 msgr2=0x7faddc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faddc0682f0 0x7faddc06a7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fadf8153770 tx=0x7fade8007a20 comp rx=0 tx=0).stop 2023-11-08T08:20:25.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 msgr2=0x7fadf8153230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf8153230 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009130 tx=0x7fadf4004750 comp rx=0 tx=0).stop 2023-11-08T08:20:25.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 shutdown_connections 2023-11-08T08:20:25.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7faddc0682f0 0x7faddc06a7a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadf815cc80 0x7fadf81f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadf806a270 0x7fadf8153230 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 --2- 172.21.15.116:0/526744127 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadf806e570 0x7fadf81f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 >> 172.21.15.116:0/526744127 conn(0x7fadf806c010 msgr2=0x7fadf80b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 shutdown_connections 2023-11-08T08:20:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.598+0000 7fae00563700 1 -- 172.21.15.116:0/526744127 wait complete. 2023-11-08T08:20:25.611 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2023-11-08T08:20:25.612 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2023-11-08T08:20:25.612 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:20:25.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2023-11-08T08:20:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.732+0000 7f19c1da2700 1 Processor -- start 2023-11-08T08:20:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19c1da2700 1 -- start start 2023-11-08T08:20:25.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19c1da2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f19bc0d0c60 0x7f19bc0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19c1da2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc160e70 0x7f19bc15c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19c1da2700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc06e900 con 0x7f19bc06e400 2023-11-08T08:20:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19c1da2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19bc160390 con 0x7f19bc0d0c60 2023-11-08T08:20:25.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19c1da2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc160670 con 0x7f19bc160e70 2023-11-08T08:20:25.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19bb7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f19bc0d0c60 0x7f19bc0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19baffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f19bc06e400 0x7f19bc0d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39840/0 (socket says 172.21.15.116:39840) 2023-11-08T08:20:25.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.733+0000 7f19baffd700 1 -- 172.21.15.116:0/2048564503 learned_addr learned my addr 172.21.15.116:0/2048564503 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:25.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19bb7fe700 1 -- 172.21.15.116:0/2048564503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc160e70 msgr2=0x7f19bc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:25.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19bb7fe700 1 --2- 172.21.15.116:0/2048564503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc160e70 0x7f19bc15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19bb7fe700 1 -- 172.21.15.116:0/2048564503 >> v1:172.21.15.116:6789/0 conn(0x7f19bc06e400 legacy=0x7f19bc0d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19bb7fe700 1 -- 172.21.15.116:0/2048564503 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc160950 con 0x7f19bc0d0c60 2023-11-08T08:20:25.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19ba7fc700 1 -- 172.21.15.116:0/2048564503 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868358393 0 0) 0x7f19bc06e900 con 0x7f19bc06e400 2023-11-08T08:20:25.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19bb7fe700 1 --2- 172.21.15.116:0/2048564503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc0b5be0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009840 tx=0x7f19b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede3bafa4c860210 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:25.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19ba7fc700 1 -- 172.21.15.116:0/2048564503 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0011830 con 0x7f19bc0d0c60 2023-11-08T08:20:25.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19ba7fc700 1 -- 172.21.15.116:0/2048564503 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b0011990 con 0x7f19bc0d0c60 2023-11-08T08:20:25.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.734+0000 7f19ba7fc700 1 -- 172.21.15.116:0/2048564503 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0011d20 con 0x7f19bc0d0c60 2023-11-08T08:20:25.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 -- 172.21.15.116:0/2048564503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 msgr2=0x7f19bc0b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 --2- 172.21.15.116:0/2048564503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc0b5be0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009840 tx=0x7f19b0009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:25.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 -- 172.21.15.116:0/2048564503 shutdown_connections 2023-11-08T08:20:25.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 --2- 172.21.15.116:0/2048564503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc0b5be0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 --2- 172.21.15.116:0/2048564503 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc160e70 0x7f19bc15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 -- 172.21.15.116:0/2048564503 >> 172.21.15.116:0/2048564503 conn(0x7f19bc06a270 msgr2=0x7f19bc06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:25.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 -- 172.21.15.116:0/2048564503 shutdown_connections 2023-11-08T08:20:25.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.735+0000 7f19c1da2700 1 -- 172.21.15.116:0/2048564503 wait complete. 2023-11-08T08:20:25.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.736+0000 7f19c1da2700 1 Processor -- start 2023-11-08T08:20:25.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.736+0000 7f19c1da2700 1 -- start start 2023-11-08T08:20:25.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc06e400 0x7f19bc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc1f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc160e70 0x7f19bc1443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0cbba0 con 0x7f19bc0d0c60 2023-11-08T08:20:25.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0b3050 con 0x7f19bc160e70 2023-11-08T08:20:25.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0b0a70 con 0x7f19bc06e400 2023-11-08T08:20:25.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc1f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc1f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39450/0 (socket says 172.21.15.116:39450) 2023-11-08T08:20:25.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 -- 172.21.15.116:0/3709434455 learned_addr learned my addr 172.21.15.116:0/3709434455 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:25.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19bb7fe700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc06e400 0x7f19bc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19bbfff700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc160e70 0x7f19bc1443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 -- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc06e400 msgr2=0x7f19bc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc06e400 0x7f19bc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 -- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc160e70 msgr2=0x7f19bc1443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:25.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc160e70 0x7f19bc1443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:25.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 -- 172.21.15.116:0/3709434455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b00094a0 con 0x7f19bc0d0c60 2023-11-08T08:20:25.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19baffd700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc1f2730 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00c540 tx=0x7f19ac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:25.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac004c80 con 0x7f19bc0d0c60 2023-11-08T08:20:25.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac004de0 con 0x7f19bc0d0c60 2023-11-08T08:20:25.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc15ceb0 con 0x7f19bc0d0c60 2023-11-08T08:20:25.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac007310 con 0x7f19bc0d0c60 2023-11-08T08:20:25.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.737+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc0d0740 con 0x7f19bc0d0c60 2023-11-08T08:20:25.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.738+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc15d040 con 0x7f19bc0d0c60 2023-11-08T08:20:25.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.739+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f19ac01a020 con 0x7f19bc0d0c60 2023-11-08T08:20:25.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.739+0000 7f19a3fff700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19a80682f0 0x7f19a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:25.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.739+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f19ac085420 con 0x7f19bc0d0c60 2023-11-08T08:20:25.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.739+0000 7f19bb7fe700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19a80682f0 0x7f19a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:25.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.740+0000 7f19bb7fe700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19a80682f0 0x7f19a806a7a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f19b00090b0 tx=0x7f19b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:25.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.744+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19ac051f80 con 0x7f19bc0d0c60 2023-11-08T08:20:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:25.995+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7f19bc0743f0 con 0x7f19bc0d0c60 2023-11-08T08:20:26.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.600+0000 7f19a3fff700 1 -- 172.21.15.116:0/3709434455 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v110) v1 ==== 152+0+0 (secure 0 0 0) 0x7f19ac0555a0 con 0x7f19bc0d0c60 2023-11-08T08:20:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19a80682f0 msgr2=0x7f19a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19a80682f0 0x7f19a806a7a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f19b00090b0 tx=0x7f19b0008030 comp rx=0 tx=0).stop 2023-11-08T08:20:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 msgr2=0x7f19bc1f2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc1f2730 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00c540 tx=0x7f19ac00c850 comp rx=0 tx=0).stop 2023-11-08T08:20:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 shutdown_connections 2023-11-08T08:20:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f19a80682f0 0x7f19a806a7a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc160e70 0x7f19bc1443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f19bc06e400 0x7f19bc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 --2- 172.21.15.116:0/3709434455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc0d0c60 0x7f19bc1f2730 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 >> 172.21.15.116:0/3709434455 conn(0x7f19bc06a270 msgr2=0x7f19bc1518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 shutdown_connections 2023-11-08T08:20:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.603+0000 7f19c1da2700 1 -- 172.21.15.116:0/3709434455 wait complete. 2023-11-08T08:20:26.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-08T08:20:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 Processor -- start 2023-11-08T08:20:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 -- start start 2023-11-08T08:20:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1cd015a4d0 0x7f1cd006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cd00cdf20 0x7f1cd00cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd015e0d0 con 0x7f1cd006a7b0 2023-11-08T08:20:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd006ac30 con 0x7f1cd00cdf20 2023-11-08T08:20:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.745+0000 7f1cd678f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd00ce310 con 0x7f1cd015a4d0 2023-11-08T08:20:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cd00cdf20 0x7f1cd00cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cd00cdf20 0x7f1cd00cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39464/0 (socket says 172.21.15.116:39464) 2023-11-08T08:20:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 -- 172.21.15.116:0/3948517958 learned_addr learned my addr 172.21.15.116:0/3948517958 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 -- 172.21.15.116:0/3948517958 >> v2:172.21.15.129:3300/0 conn(0x7f1cd015a4d0 msgr2=0x7f1cd006a270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:20:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 --2- 172.21.15.116:0/3948517958 >> v2:172.21.15.129:3300/0 conn(0x7f1cd015a4d0 0x7f1cd006a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 -- 172.21.15.116:0/3948517958 >> v1:172.21.15.129:6789/0 conn(0x7f1cd006a7b0 legacy=0x7f1cd00cd810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-08T08:20:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 -- 172.21.15.116:0/3948517958 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd00ce560 con 0x7f1cd00cdf20 2023-11-08T08:20:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cd4d2c700 1 --2- 172.21.15.116:0/3948517958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd00cdf20 0x7f1cd00cb010 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1cd006b140 tx=0x7f1cc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d2f8884e2e03a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cceffd700 1 -- 172.21.15.116:0/3948517958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc800e070 con 0x7f1cd00cdf20 2023-11-08T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cceffd700 1 -- 172.21.15.116:0/3948517958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc8003a20 con 0x7f1cd00cdf20 2023-11-08T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.746+0000 7f1cceffd700 1 -- 172.21.15.116:0/3948517958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8003dd0 con 0x7f1cd00cdf20 2023-11-08T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 -- 172.21.15.116:0/3948517958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd00cdf20 msgr2=0x7f1cd00cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3948517958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd00cdf20 0x7f1cd00cb010 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1cd006b140 tx=0x7f1cc800b330 comp rx=0 tx=0).stop 2023-11-08T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 -- 172.21.15.116:0/3948517958 shutdown_connections 2023-11-08T08:20:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3948517958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd00cdf20 0x7f1cd00cb010 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3948517958 >> v2:172.21.15.129:3300/0 conn(0x7f1cd015a4d0 0x7f1cd006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 -- 172.21.15.116:0/3948517958 >> 172.21.15.116:0/3948517958 conn(0x7f1cd006c010 msgr2=0x7f1cd00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 -- 172.21.15.116:0/3948517958 shutdown_connections 2023-11-08T08:20:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.747+0000 7f1cd678f700 1 -- 172.21.15.116:0/3948517958 wait complete. 2023-11-08T08:20:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.748+0000 7f1cd678f700 1 Processor -- start 2023-11-08T08:20:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.748+0000 7f1cd678f700 1 -- start start 2023-11-08T08:20:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1cd678f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 0x7f1cd014ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1cd678f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1cd00cdf20 0x7f1cd01f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1cd678f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cd015a4d0 0x7f1cd01e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1cd678f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd01e4100 con 0x7f1cd006a7b0 2023-11-08T08:20:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1cd678f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd01e3b10 con 0x7f1cd015a4d0 2023-11-08T08:20:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1cd678f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd01f49c0 con 0x7f1cd00cdf20 2023-11-08T08:20:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1ccffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 0x7f1cd014ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1ccffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 0x7f1cd014ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39480/0 (socket says 172.21.15.116:39480) 2023-11-08T08:20:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.749+0000 7f1ccffff700 1 -- 172.21.15.116:0/3919250844 learned_addr learned my addr 172.21.15.116:0/3919250844 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1ccffff700 1 -- 172.21.15.116:0/3919250844 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1cd00cdf20 msgr2=0x7f1cd01f3f00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:20:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1ccffff700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1cd00cdf20 0x7f1cd01f3f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1ccffff700 1 -- 172.21.15.116:0/3919250844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cd015a4d0 msgr2=0x7f1cd01e5fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-08T08:20:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1ccffff700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cd015a4d0 0x7f1cd01e5fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1ccffff700 1 -- 172.21.15.116:0/3919250844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc8009b30 con 0x7f1cd006a7b0 2023-11-08T08:20:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1ccffff700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 0x7f1cd014ee10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400b470 tx=0x7f1cc400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4011830 con 0x7f1cd006a7b0 2023-11-08T08:20:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc4011990 con 0x7f1cd006a7b0 2023-11-08T08:20:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd015e600 con 0x7f1cd006a7b0 2023-11-08T08:20:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4011d40 con 0x7f1cd006a7b0 2023-11-08T08:20:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.750+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd015e320 con 0x7f1cd006a7b0 2023-11-08T08:20:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.751+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd015e8e0 con 0x7f1cd006a7b0 2023-11-08T08:20:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.751+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1cc4021070 con 0x7f1cd006a7b0 2023-11-08T08:20:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.752+0000 7f1cccff9700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1cbc0682f0 0x7f1cbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.752+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f1cc40858b0 con 0x7f1cd006a7b0 2023-11-08T08:20:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.752+0000 7f1ccf7fe700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1cbc0682f0 0x7f1cbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.752+0000 7f1ccf7fe700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1cbc0682f0 0x7f1cbc06a7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00b43e0 tx=0x7f1cc0006c60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.756+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1cc4052410 con 0x7f1cd006a7b0 2023-11-08T08:20:26.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.939+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1cd00743f0 con 0x7f1cd006a7b0 2023-11-08T08:20:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.945+0000 7f1cccff9700 1 -- 172.21.15.116:0/3919250844 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1cc4015020 con 0x7f1cd006a7b0 2023-11-08T08:20:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1cbc0682f0 msgr2=0x7f1cbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1cbc0682f0 0x7f1cbc06a7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00b43e0 tx=0x7f1cc0006c60 comp rx=0 tx=0).stop 2023-11-08T08:20:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 msgr2=0x7f1cd014ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 0x7f1cd014ee10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400b470 tx=0x7f1cc400b830 comp rx=0 tx=0).stop 2023-11-08T08:20:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 shutdown_connections 2023-11-08T08:20:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1cbc0682f0 0x7f1cbc06a7a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cd015a4d0 0x7f1cd01e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd006a7b0 0x7f1cd014ee10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 --2- 172.21.15.116:0/3919250844 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1cd00cdf20 0x7f1cd01f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 >> 172.21.15.116:0/3919250844 conn(0x7f1cd006c010 msgr2=0x7f1cd00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 shutdown_connections 2023-11-08T08:20:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:26.947+0000 7f1cd678f700 1 -- 172.21.15.116:0/3919250844 wait complete. 2023-11-08T08:20:26.960 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-08T08:20:26.961 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:20:27.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:20:27.051 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-08T08:20:27.051 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:20:27.051 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:27.070 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:20:27.070 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:20:27.071 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:27.091 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:20:27.091 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:20:27.092 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:27.128 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:20:27.128 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:20:27.129 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:20:27.150 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:20:27.151 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-08T08:20:27.151 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.285+0000 7f71cb85e700 1 Processor -- start 2023-11-08T08:20:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.285+0000 7f71cb85e700 1 -- start start 2023-11-08T08:20:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71cb85e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f71c406ab10 0x7f71c406aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71cb85e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c40cd5a0 0x7f71c4160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71cb85e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71c40b22a0 0x7f71c4160840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71cb85e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40d1ee0 con 0x7f71c40b22a0 2023-11-08T08:20:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71cb85e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c406b420 con 0x7f71c406ab10 2023-11-08T08:20:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71cb85e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40cdb10 con 0x7f71c40cd5a0 2023-11-08T08:20:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71c9dfb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71c40b22a0 0x7f71c4160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.286+0000 7f71c9dfb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71c40b22a0 0x7f71c4160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39490/0 (socket says 172.21.15.116:39490) 2023-11-08T08:20:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 -- 172.21.15.116:0/1139247597 learned_addr learned my addr 172.21.15.116:0/1139247597 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c95fa700 1 --2- 172.21.15.116:0/1139247597 >> v2:172.21.15.129:3300/0 conn(0x7f71c406ab10 0x7f71c406aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c8df9700 1 --2- 172.21.15.116:0/1139247597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c40cd5a0 0x7f71c4160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 -- 172.21.15.116:0/1139247597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c40cd5a0 msgr2=0x7f71c4160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 --2- 172.21.15.116:0/1139247597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c40cd5a0 0x7f71c4160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 -- 172.21.15.116:0/1139247597 >> v2:172.21.15.129:3300/0 conn(0x7f71c406ab10 msgr2=0x7f71c406aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 --2- 172.21.15.116:0/1139247597 >> v2:172.21.15.129:3300/0 conn(0x7f71c406ab10 0x7f71c406aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 -- 172.21.15.116:0/1139247597 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c40d1890 con 0x7f71c40b22a0 2023-11-08T08:20:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c95fa700 1 --2- 172.21.15.116:0/1139247597 >> v2:172.21.15.129:3300/0 conn(0x7f71c406ab10 0x7f71c406aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71c9dfb700 1 --2- 172.21.15.116:0/1139247597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c4160840 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f71c40d0cb0 tx=0x7f71bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f2f7fec3253efb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71bbfff700 1 -- 172.21.15.116:0/1139247597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00e070 con 0x7f71c40b22a0 2023-11-08T08:20:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71bbfff700 1 -- 172.21.15.116:0/1139247597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc00b9c0 con 0x7f71c40b22a0 2023-11-08T08:20:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.287+0000 7f71bbfff700 1 -- 172.21.15.116:0/1139247597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00bd70 con 0x7f71c40b22a0 2023-11-08T08:20:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 -- 172.21.15.116:0/1139247597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 msgr2=0x7f71c4160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1139247597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c4160840 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f71c40d0cb0 tx=0x7f71bc00b330 comp rx=0 tx=0).stop 2023-11-08T08:20:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 -- 172.21.15.116:0/1139247597 shutdown_connections 2023-11-08T08:20:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1139247597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c4160840 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1139247597 >> v2:172.21.15.129:3300/0 conn(0x7f71c406ab10 0x7f71c406aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1139247597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c40cd5a0 0x7f71c4160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 -- 172.21.15.116:0/1139247597 >> 172.21.15.116:0/1139247597 conn(0x7f71c406c520 msgr2=0x7f71c406c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 -- 172.21.15.116:0/1139247597 shutdown_connections 2023-11-08T08:20:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.288+0000 7f71cb85e700 1 -- 172.21.15.116:0/1139247597 wait complete. 2023-11-08T08:20:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.289+0000 7f71cb85e700 1 Processor -- start 2023-11-08T08:20:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.289+0000 7f71cb85e700 1 -- start start 2023-11-08T08:20:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c406ab10 0x7f71c40cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c41483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71c40cd5a0 0x7f71c40c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c4148910 con 0x7f71c40b22a0 2023-11-08T08:20:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c7ee0 con 0x7f71c40cd5a0 2023-11-08T08:20:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c72d0 con 0x7f71c406ab10 2023-11-08T08:20:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c41483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c41483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39492/0 (socket says 172.21.15.116:39492) 2023-11-08T08:20:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 -- 172.21.15.116:0/1794216256 learned_addr learned my addr 172.21.15.116:0/1794216256 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c95fa700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c406ab10 0x7f71c40cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c9dfb700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71c40cd5a0 0x7f71c40c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 -- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c406ab10 msgr2=0x7f71c40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c406ab10 0x7f71c40cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 -- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71c40cd5a0 msgr2=0x7f71c40c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71c40cd5a0 0x7f71c40c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 -- 172.21.15.116:0/1794216256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc009b30 con 0x7f71c40b22a0 2023-11-08T08:20:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71c8df9700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c41483d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c7f0 tx=0x7f71c000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0011800 con 0x7f71c40b22a0 2023-11-08T08:20:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0011960 con 0x7f71c40b22a0 2023-11-08T08:20:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c415cad0 con 0x7f71c40b22a0 2023-11-08T08:20:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.290+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0011cf0 con 0x7f71c40b22a0 2023-11-08T08:20:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.291+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c40d1b70 con 0x7f71c40b22a0 2023-11-08T08:20:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.292+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c40b1ec0 con 0x7f71c40b22a0 2023-11-08T08:20:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.292+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f71c0021070 con 0x7f71c40b22a0 2023-11-08T08:20:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.292+0000 7f71b9ffb700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71a80682f0 0x7f71a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.292+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f71c00858c0 con 0x7f71c40b22a0 2023-11-08T08:20:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.292+0000 7f71c95fa700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71a80682f0 0x7f71a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.293+0000 7f71c95fa700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71a80682f0 0x7f71a806a7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f71b4006fd0 tx=0x7f71b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.296+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f71c00523a0 con 0x7f71c40b22a0 2023-11-08T08:20:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.518+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71c40c8050 con 0x7f71c40b22a0 2023-11-08T08:20:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.519+0000 7f71b9ffb700 1 -- 172.21.15.116:0/1794216256 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3538 (secure 0 0 0) 0x7f71c001f020 con 0x7f71c40b22a0 2023-11-08T08:20:27.521 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:27.521 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4745,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1472524126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1472524126},{"type":"v1","addr":"172.21.15.129:6834","nonce":1472524126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2023560261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2023560261},{"type":"v1","addr":"172.21.15.129:6835","nonce":2023560261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3833907634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3833907634},{"type":"v1","addr":"172.21.15.116:6836","nonce":3833907634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5697,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/642841892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":642841892},{"type":"v1","addr":"172.21.15.116:6837","nonce":642841892}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2023-11-08T08:20:27.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71a80682f0 msgr2=0x7f71a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71a80682f0 0x7f71a806a7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f71b4006fd0 tx=0x7f71b4008040 comp rx=0 tx=0).stop 2023-11-08T08:20:27.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 msgr2=0x7f71c41483d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c41483d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c7f0 tx=0x7f71c000cbb0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 shutdown_connections 2023-11-08T08:20:27.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71a80682f0 0x7f71a806a7a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71c40cd5a0 0x7f71c40c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71c406ab10 0x7f71c40cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 --2- 172.21.15.116:0/1794216256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c40b22a0 0x7f71c41483d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 >> 172.21.15.116:0/1794216256 conn(0x7f71c406c520 msgr2=0x7f71c40b72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:27.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 shutdown_connections 2023-11-08T08:20:27.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.521+0000 7f71cb85e700 1 -- 172.21.15.116:0/1794216256 wait complete. 2023-11-08T08:20:27.525 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 18 2023-11-08T08:20:27.534 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-08T08:20:27.535 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-08T08:20:27.535 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-08T08:20:27.535 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:20:27.536 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-08T08:20:27.536 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-08T08:20:27.536 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-08T08:20:27.537 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:20:27.537 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-08T08:20:27.537 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-08T08:20:27.537 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-08T08:20:27.538 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:20:27.538 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-08T08:20:27.538 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-08T08:20:27.539 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-08T08:20:27.539 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:20:27.544 INFO:tasks.ceph.mds.a:Stopped 2023-11-08T08:20:27.544 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-08T08:20:27.545 INFO:tasks.ceph.mds.c:Stopped 2023-11-08T08:20:27.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-08T08:20:27.552 INFO:tasks.ceph.mds.b:Stopped 2023-11-08T08:20:27.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-08T08:20:27.553 INFO:tasks.ceph.mds.d:Stopped 2023-11-08T08:20:27.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-08T08:20:27.589 INFO:tasks.ceph.mds.a:Started 2023-11-08T08:20:27.590 DEBUG:teuthology.parallel:result is None 2023-11-08T08:20:27.591 INFO:tasks.ceph.mds.c:Started 2023-11-08T08:20:27.592 DEBUG:teuthology.parallel:result is None 2023-11-08T08:20:27.596 INFO:tasks.ceph.mds.b:Started 2023-11-08T08:20:27.596 DEBUG:teuthology.parallel:result is None 2023-11-08T08:20:27.598 INFO:tasks.ceph.mds.d:Started 2023-11-08T08:20:27.598 DEBUG:teuthology.parallel:result is None 2023-11-08T08:20:27.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-08T08:20:27.668 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-08T08:20:27.669 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-08T08:20:27.685 INFO:tasks.ceph.mds.d.smithi129.stdout:starting mds.d at 2023-11-08T08:20:27.685 INFO:tasks.ceph.mds.b.smithi129.stdout:starting mds.b at 2023-11-08T08:20:27.686 INFO:tasks.ceph.mds.d.smithi129.stderr:2023-11-08T08:20:27.686+0000 7f018cc1a700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6834/0: (98) Address already in use 2023-11-08T08:20:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.728+0000 7fe572ea7700 1 Processor -- start 2023-11-08T08:20:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.728+0000 7fe572ea7700 1 -- start start 2023-11-08T08:20:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe572ea7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe572ea7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe56c06b220 0x7fe56c0d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe572ea7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c0d2d30 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe572ea7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c0d0b40 con 0x7fe56c06b220 2023-11-08T08:20:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe572ea7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c1611f0 con 0x7fe56c15c8c0 2023-11-08T08:20:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe571444700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe56c06b220 0x7fe56c0d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe56bfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe56c0b5dc0 0x7fe56c06ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39862/0 (socket says 172.21.15.116:39862) 2023-11-08T08:20:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.729+0000 7fe56bfff700 1 -- 172.21.15.116:0/1368131337 learned_addr learned my addr 172.21.15.116:0/1368131337 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe571444700 1 -- 172.21.15.116:0/1368131337 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 msgr2=0x7fe56c160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe571444700 1 --2- 172.21.15.116:0/1368131337 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe571444700 1 -- 172.21.15.116:0/1368131337 >> v1:172.21.15.116:6789/0 conn(0x7fe56c0b5dc0 legacy=0x7fe56c06ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe571444700 1 -- 172.21.15.116:0/1368131337 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0d2a50 con 0x7fe56c06b220 2023-11-08T08:20:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe571444700 1 --2- 172.21.15.116:0/1368131337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c06b220 0x7fe56c0d0fd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009840 tx=0x7fe55c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d0e2b61c686981 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe56b7fe700 1 -- 172.21.15.116:0/1368131337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c011830 con 0x7fe56c06b220 2023-11-08T08:20:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe56b7fe700 1 -- 172.21.15.116:0/1368131337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c011990 con 0x7fe56c06b220 2023-11-08T08:20:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe56b7fe700 1 -- 172.21.15.116:0/1368131337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c011d20 con 0x7fe56c06b220 2023-11-08T08:20:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 -- 172.21.15.116:0/1368131337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c06b220 msgr2=0x7fe56c0d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 --2- 172.21.15.116:0/1368131337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c06b220 0x7fe56c0d0fd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009840 tx=0x7fe55c009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 -- 172.21.15.116:0/1368131337 shutdown_connections 2023-11-08T08:20:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 --2- 172.21.15.116:0/1368131337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c06b220 0x7fe56c0d0fd0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 --2- 172.21.15.116:0/1368131337 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 -- 172.21.15.116:0/1368131337 >> 172.21.15.116:0/1368131337 conn(0x7fe56c06c520 msgr2=0x7fe56c06c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.730+0000 7fe572ea7700 1 -- 172.21.15.116:0/1368131337 shutdown_connections 2023-11-08T08:20:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.731+0000 7fe572ea7700 1 -- 172.21.15.116:0/1368131337 wait complete. 2023-11-08T08:20:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.732+0000 7fe572ea7700 1 Processor -- start 2023-11-08T08:20:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.732+0000 7fe572ea7700 1 -- start start 2023-11-08T08:20:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.732+0000 7fe572ea7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c06b220 0x7fe56c154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 0x7fe56c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c0cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0c8ca0 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0c8f80 con 0x7fe56c06b220 2023-11-08T08:20:27.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0c9260 con 0x7fe56c15c8c0 2023-11-08T08:20:27.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 0x7fe56c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 0x7fe56c0cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39530/0 (socket says 172.21.15.116:39530) 2023-11-08T08:20:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 -- 172.21.15.116:0/2025938788 learned_addr learned my addr 172.21.15.116:0/2025938788 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe571444700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c0cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe570c43700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c06b220 0x7fe56c154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 -- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 msgr2=0x7fe56c0cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c0cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 -- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c06b220 msgr2=0x7fe56c154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c06b220 0x7fe56c154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 -- 172.21.15.116:0/2025938788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c0094a0 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe56bfff700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 0x7fe56c0cb0b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe55400c520 tx=0x7fe55400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe554005d70 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5540070d0 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c0cda40 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe554007440 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.733+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c0cd760 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.734+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c0d2700 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.735+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe554019030 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.735+0000 7fe5697fa700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe54c0682f0 0x7fe54c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:27.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.735+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fe554084a80 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.735+0000 7fe570c43700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe54c0682f0 0x7fe54c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:27.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.735+0000 7fe570c43700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe54c0682f0 0x7fe54c06a7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe564009f20 tx=0x7fe5640094f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:27.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.739+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe55404b5e0 con 0x7fe56c0b5dc0 2023-11-08T08:20:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:27.936+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe56c067d70 con 0x7fe56c0b5dc0 2023-11-08T08:20:28.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.718+0000 7fe5697fa700 1 -- 172.21.15.116:0/2025938788 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v40) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe554017080 con 0x7fe56c0b5dc0 2023-11-08T08:20:28.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe54c0682f0 msgr2=0x7fe54c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:28.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe54c0682f0 0x7fe54c06a7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe564009f20 tx=0x7fe5640094f0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 msgr2=0x7fe56c0cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:28.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 0x7fe56c0cb0b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe55400c520 tx=0x7fe55400c8e0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 shutdown_connections 2023-11-08T08:20:28.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c15c8c0 0x7fe56c0cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe54c0682f0 0x7fe54c06a7a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c06b220 0x7fe56c154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 --2- 172.21.15.116:0/2025938788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c0b5dc0 0x7fe56c0cb0b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 >> 172.21.15.116:0/2025938788 conn(0x7fe56c06c520 msgr2=0x7fe56c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:28.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 shutdown_connections 2023-11-08T08:20:28.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.720+0000 7fe572ea7700 1 -- 172.21.15.116:0/2025938788 wait complete. 2023-11-08T08:20:28.727 INFO:teuthology.orchestra.run.smithi116.stderr: removed all blocklist entries 2023-11-08T08:20:28.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-08T08:20:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.857+0000 7f073fd02700 1 Processor -- start 2023-11-08T08:20:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.857+0000 7f073fd02700 1 -- start start 2023-11-08T08:20:28.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073fd02700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07380d0c60 0x7f07380b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073fd02700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f073806e400 0x7f07380d1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073fd02700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0738160e70 0x7f073806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073fd02700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07380d1570 con 0x7f073806e400 2023-11-08T08:20:28.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073fd02700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073806e900 con 0x7f0738160e70 2023-11-08T08:20:28.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073fd02700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0738160390 con 0x7f07380d0c60 2023-11-08T08:20:28.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073d29d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f073806e400 0x7f07380d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:28.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073d29d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f073806e400 0x7f07380d1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39536/0 (socket says 172.21.15.116:39536) 2023-11-08T08:20:28.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.858+0000 7f073d29d700 1 -- 172.21.15.116:0/1819817856 learned_addr learned my addr 172.21.15.116:0/1819817856 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:28.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073d29d700 1 -- 172.21.15.116:0/1819817856 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07380d0c60 msgr2=0x7f07380b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:28.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073d29d700 1 --2- 172.21.15.116:0/1819817856 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07380d0c60 0x7f07380b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073d29d700 1 -- 172.21.15.116:0/1819817856 >> v2:172.21.15.129:3300/0 conn(0x7f0738160e70 msgr2=0x7f073806f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073e29f700 1 --2- 172.21.15.116:0/1819817856 >> v2:172.21.15.129:3300/0 conn(0x7f0738160e70 0x7f073806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073d29d700 1 --2- 172.21.15.116:0/1819817856 >> v2:172.21.15.129:3300/0 conn(0x7f0738160e70 0x7f073806f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073d29d700 1 -- 172.21.15.116:0/1819817856 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0738160670 con 0x7f073806e400 2023-11-08T08:20:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073e29f700 1 --2- 172.21.15.116:0/1819817856 >> v2:172.21.15.129:3300/0 conn(0x7f0738160e70 0x7f073806f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073d29d700 1 --2- 172.21.15.116:0/1819817856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f073806e400 0x7f07380d1030 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0738160130 tx=0x7f072c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea6d2e0fa26b46 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073ca9c700 1 -- 172.21.15.116:0/1819817856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c00b870 con 0x7f073806e400 2023-11-08T08:20:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073ca9c700 1 -- 172.21.15.116:0/1819817856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c00b9d0 con 0x7f073806e400 2023-11-08T08:20:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073ca9c700 1 -- 172.21.15.116:0/1819817856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c00bd60 con 0x7f073806e400 2023-11-08T08:20:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073fd02700 1 -- 172.21.15.116:0/1819817856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f073806e400 msgr2=0x7f07380d1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.859+0000 7f073fd02700 1 --2- 172.21.15.116:0/1819817856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f073806e400 0x7f07380d1030 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0738160130 tx=0x7f072c009bd0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 -- 172.21.15.116:0/1819817856 shutdown_connections 2023-11-08T08:20:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 --2- 172.21.15.116:0/1819817856 >> v2:172.21.15.129:3300/0 conn(0x7f0738160e70 0x7f073806f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 --2- 172.21.15.116:0/1819817856 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07380d0c60 0x7f07380b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 --2- 172.21.15.116:0/1819817856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f073806e400 0x7f07380d1030 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 -- 172.21.15.116:0/1819817856 >> 172.21.15.116:0/1819817856 conn(0x7f073806a270 msgr2=0x7f073806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 -- 172.21.15.116:0/1819817856 shutdown_connections 2023-11-08T08:20:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.860+0000 7f073fd02700 1 -- 172.21.15.116:0/1819817856 wait complete. 2023-11-08T08:20:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.861+0000 7f073fd02700 1 Processor -- start 2023-11-08T08:20:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.861+0000 7f073fd02700 1 -- start start 2023-11-08T08:20:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073fd02700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f073806e400 0x7f0738154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073fd02700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 0x7f07381f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073fd02700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0738160e70 0x7f07381f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073fd02700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0738155710 con 0x7f07380d0c60 2023-11-08T08:20:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073fd02700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07381559f0 con 0x7f073806e400 2023-11-08T08:20:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073fd02700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0738155cd0 con 0x7f0738160e70 2023-11-08T08:20:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 0x7f07381f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 0x7f07381f5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39542/0 (socket says 172.21.15.116:39542) 2023-11-08T08:20:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 -- 172.21.15.116:0/10922363 learned_addr learned my addr 172.21.15.116:0/10922363 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073e29f700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0738160e70 0x7f07381f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073da9e700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f073806e400 0x7f0738154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 -- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0738160e70 msgr2=0x7f07381f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0738160e70 0x7f07381f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 -- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f073806e400 msgr2=0x7f0738154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f073806e400 0x7f0738154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073d29d700 1 -- 172.21.15.116:0/10922363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c009550 con 0x7f07380d0c60 2023-11-08T08:20:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073da9e700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f073806e400 0x7f0738154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.862+0000 7f073e29f700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0738160e70 0x7f07381f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.863+0000 7f073d29d700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 0x7f07381f5b30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f072c013000 tx=0x7f072c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.863+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c004310 con 0x7f07380d0c60 2023-11-08T08:20:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.863+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c01f070 con 0x7f07380d0c60 2023-11-08T08:20:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.863+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c0036a0 con 0x7f07380d0c60 2023-11-08T08:20:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.863+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07380d0740 con 0x7f07380d0c60 2023-11-08T08:20:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.863+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0738160950 con 0x7f07380d0c60 2023-11-08T08:20:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.864+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073806fbf0 con 0x7f07380d0c60 2023-11-08T08:20:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.864+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f072c02a020 con 0x7f07380d0c60 2023-11-08T08:20:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.864+0000 7f07367fc700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07240682f0 0x7f072406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.865+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f072c087060 con 0x7f07380d0c60 2023-11-08T08:20:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.865+0000 7f073da9e700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07240682f0 0x7f072406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:28.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.865+0000 7f073da9e700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07240682f0 0x7f072406a7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0730007900 tx=0x7f0730008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:28.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:28.869+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f072c0b85e0 con 0x7f07380d0c60 2023-11-08T08:20:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.123+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f07380743f0 con 0x7f07380d0c60 2023-11-08T08:20:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.123+0000 7f07367fc700 1 -- 172.21.15.116:0/10922363 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f072c053b20 con 0x7f07380d0c60 2023-11-08T08:20:29.125 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:29.125 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:20:29.125 INFO:teuthology.orchestra.run.smithi116.stdout: "auth_dump": [ 2023-11-08T08:20:29.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.126 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.a", 2023-11-08T08:20:29.126 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlwPmvARAAdJKHnQqKVPWnN7uQiHc+WQ==", 2023-11-08T08:20:29.126 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.126 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.127 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:20:29.127 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:20:29.127 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.128 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.b", 2023-11-08T08:20:29.128 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlXUbqDRAATJnFxEiS//echgEFyiS+iw==", 2023-11-08T08:20:29.128 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.128 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.129 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:20:29.129 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:20:29.129 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.130 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.c", 2023-11-08T08:20:29.130 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlNA7+CBAAkUCXKhY1qacysaUprxlU9w==", 2023-11-08T08:20:29.130 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.130 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.131 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:20:29.131 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:20:29.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.132 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.d", 2023-11-08T08:20:29.132 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtltlgoFRAAwAkFfezamgYHcC9awLpLpA==", 2023-11-08T08:20:29.132 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.132 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:20:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:20:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.0", 2023-11-08T08:20:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAyREtlwCT6MBAA0hUY6XxZE8Q/cW66704LAA==", 2023-11-08T08:20:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.1", 2023-11-08T08:20:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA1REtlHxOBLxAAHR12A9jsXL7W8TeG55VeWg==", 2023-11-08T08:20:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.2", 2023-11-08T08:20:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA4REtlcPidLhAAhNEoyldnbVRtP9IQr08wag==", 2023-11-08T08:20:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.139 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.3", 2023-11-08T08:20:29.139 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA7REtlH/3XLBAASJ532Or0yrTC8qfr3kCziQ==", 2023-11-08T08:20:29.140 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.140 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.140 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.140 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.141 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.4", 2023-11-08T08:20:29.141 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBBREtl5+beLxAAXM67eNV2/FzHU4+gp3zMng==", 2023-11-08T08:20:29.141 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.143 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.5", 2023-11-08T08:20:29.143 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBEREtl4pChLhAA470kPHr4WFBGgwEi0CB95A==", 2023-11-08T08:20:29.143 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.143 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.144 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.144 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.145 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.6", 2023-11-08T08:20:29.145 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBHREtlh5g/LRAAY7eRgyAdWK0ExY6Qaf6hgQ==", 2023-11-08T08:20:29.145 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.145 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.145 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.146 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.146 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.7", 2023-11-08T08:20:29.147 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBKREtly78MKxAA5+d+0MekiMWQK1sAn1Ue6w==", 2023-11-08T08:20:29.147 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.147 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:20:29.147 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:20:29.147 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.148 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.0", 2023-11-08T08:20:29.148 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlHhh+GRAAKHMDRBv0YayhqERZiWXGAw==", 2023-11-08T08:20:29.149 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.149 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.149 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow r", 2023-11-08T08:20:29.149 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rw pool=cephfs_data" 2023-11-08T08:20:29.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.150 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.1", 2023-11-08T08:20:29.150 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtl2ICCIhAAdgVi5BOXs1xqGLKgTLEXoQ==", 2023-11-08T08:20:29.150 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.151 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.151 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:20:29.151 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:20:29.151 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:20:29.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.152 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.2", 2023-11-08T08:20:29.152 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlKh8kHxAA/Ear8UA/wixquw0KBzet2A==", 2023-11-08T08:20:29.152 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.152 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.153 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:20:29.153 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:20:29.153 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:20:29.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.154 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.3", 2023-11-08T08:20:29.154 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtl17vQJRAA/R0BvGRHgw8ZySwL4KyuGg==", 2023-11-08T08:20:29.154 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.154 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:20:29.155 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:20:29.155 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:20:29.155 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:20:29.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.156 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.admin", 2023-11-08T08:20:29.156 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtlwc+XHhAA8ruR6iJNLTqLSXROZ9iRHQ==", 2023-11-08T08:20:29.156 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.156 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:20:29.157 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:20:29.157 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:20:29.157 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.157 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.158 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.x", 2023-11-08T08:20:29.158 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtl1TH6ORAAQMEd7GQJFqk28L6HfaYPlg==", 2023-11-08T08:20:29.158 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.158 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:20:29.159 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-08T08:20:29.159 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:29.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:29.160 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.y", 2023-11-08T08:20:29.160 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtlXRIMNBAAC9TkKhc8QzTE+DK2eDs/bg==", 2023-11-08T08:20:29.160 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:20:29.160 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:20:29.160 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-08T08:20:29.160 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:20:29.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:29.161 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-11-08T08:20:29.161 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:20:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07240682f0 msgr2=0x7f072406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07240682f0 0x7f072406a7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0730007900 tx=0x7f0730008040 comp rx=0 tx=0).stop 2023-11-08T08:20:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 msgr2=0x7f07381f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 0x7f07381f5b30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f072c013000 tx=0x7f072c004750 comp rx=0 tx=0).stop 2023-11-08T08:20:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 shutdown_connections 2023-11-08T08:20:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0738160e70 0x7f07381f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07240682f0 0x7f072406a7a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.125+0000 7f073fd02700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f073806e400 0x7f0738154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.126+0000 7f073fd02700 1 --2- 172.21.15.116:0/10922363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07380d0c60 0x7f07381f5b30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.126+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 >> 172.21.15.116:0/10922363 conn(0x7f073806a270 msgr2=0x7f07381527c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.126+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 shutdown_connections 2023-11-08T08:20:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.126+0000 7f073fd02700 1 -- 172.21.15.116:0/10922363 wait complete. 2023-11-08T08:20:29.166 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.286+0000 7ff3bbe74700 1 Processor -- start 2023-11-08T08:20:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.286+0000 7ff3bbe74700 1 -- start start 2023-11-08T08:20:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3bbe74700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff3b415e010 0x7ff3b40ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3bbe74700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b415af30 con 0x7ff3b40d15f0 2023-11-08T08:20:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3bbe74700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b415ef40 con 0x7ff3b406e400 2023-11-08T08:20:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3bbe74700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b415a9d0 con 0x7ff3b415e010 2023-11-08T08:20:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3ba411700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff3b415e010 0x7ff3b40ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3b940f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff3b40d15f0 0x7ff3b415b3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39876/0 (socket says 172.21.15.116:39876) 2023-11-08T08:20:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3b940f700 1 -- 172.21.15.116:0/2525268628 learned_addr learned my addr 172.21.15.116:0/2525268628 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3ba411700 1 -- 172.21.15.116:0/2525268628 >> v1:172.21.15.129:6789/0 conn(0x7ff3b406e400 legacy=0x7ff3b406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3ba411700 1 -- 172.21.15.116:0/2525268628 >> v1:172.21.15.116:6789/0 conn(0x7ff3b40d15f0 legacy=0x7ff3b415b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3ba411700 1 -- 172.21.15.116:0/2525268628 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b415e980 con 0x7ff3b415e010 2023-11-08T08:20:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3ba411700 1 --2- 172.21.15.116:0/2525268628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b40ce970 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009d90 tx=0x7ff3ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dfb762138245f20 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.287+0000 7ff3b8c0e700 1 -- 172.21.15.116:0/2525268628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00e070 con 0x7ff3b415e010 2023-11-08T08:20:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3b8c0e700 1 -- 172.21.15.116:0/2525268628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac00bb90 con 0x7ff3b415e010 2023-11-08T08:20:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3b8c0e700 1 -- 172.21.15.116:0/2525268628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac003a20 con 0x7ff3b415e010 2023-11-08T08:20:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2525268628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 msgr2=0x7ff3b40ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2525268628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b40ce970 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009d90 tx=0x7ff3ac00b330 comp rx=0 tx=0).stop 2023-11-08T08:20:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2525268628 shutdown_connections 2023-11-08T08:20:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2525268628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b40ce970 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2525268628 >> 172.21.15.116:0/2525268628 conn(0x7ff3b406a270 msgr2=0x7ff3b406a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2525268628 shutdown_connections 2023-11-08T08:20:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.288+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2525268628 wait complete. 2023-11-08T08:20:29.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.289+0000 7ff3bbe74700 1 Processor -- start 2023-11-08T08:20:29.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.289+0000 7ff3bbe74700 1 -- start start 2023-11-08T08:20:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3bbe74700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff3b406e400 0x7ff3b4148c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3bbe74700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3b40d15f0 0x7ff3b4151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3bbe74700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b4154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3bbe74700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4158680 con 0x7ff3b415e010 2023-11-08T08:20:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3bbe74700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4158960 con 0x7ff3b40d15f0 2023-11-08T08:20:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3bbe74700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4158c40 con 0x7ff3b406e400 2023-11-08T08:20:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b4154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b4154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39556/0 (socket says 172.21.15.116:39556) 2023-11-08T08:20:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 -- 172.21.15.116:0/2005337790 learned_addr learned my addr 172.21.15.116:0/2005337790 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3b940f700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3b40d15f0 0x7ff3b4151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3b9c10700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff3b406e400 0x7ff3b4148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 -- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff3b406e400 msgr2=0x7ff3b4148c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff3b406e400 0x7ff3b4148c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 -- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3b40d15f0 msgr2=0x7ff3b4151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3b40d15f0 0x7ff3b4151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.290+0000 7ff3ba411700 1 -- 172.21.15.116:0/2005337790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac009a70 con 0x7ff3b415e010 2023-11-08T08:20:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.291+0000 7ff3ba411700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b4154ee0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40b6eb0 tx=0x7ff3ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.291+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac012630 con 0x7ff3b415e010 2023-11-08T08:20:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.291+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac0127b0 con 0x7ff3b415e010 2023-11-08T08:20:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.291+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b415abc0 con 0x7ff3b415e010 2023-11-08T08:20:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.291+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac012b60 con 0x7ff3b415e010 2023-11-08T08:20:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.291+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b415ec60 con 0x7ff3b415e010 2023-11-08T08:20:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.292+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b415e490 con 0x7ff3b415e010 2023-11-08T08:20:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.292+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff3ac02c070 con 0x7ff3b415e010 2023-11-08T08:20:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.293+0000 7ff3aa7fc700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff3980682f0 0x7ff39806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.293+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff3ac087f00 con 0x7ff3b415e010 2023-11-08T08:20:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.293+0000 7ff3b9c10700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff3980682f0 0x7ff39806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.293+0000 7ff3b9c10700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff3980682f0 0x7ff39806a7a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009870 tx=0x7ff3b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.297+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff3ac054b50 con 0x7ff3b415e010 2023-11-08T08:20:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.478+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3b40743f0 con 0x7ff3b415e010 2023-11-08T08:20:29.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.479+0000 7ff3aa7fc700 1 -- 172.21.15.116:0/2005337790 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+14257 (secure 0 0 0) 0x7ff3ac058170 con 0x7ff3b415e010 2023-11-08T08:20:29.480 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:29.481 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:28.712028+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:29.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff3980682f0 msgr2=0x7ff39806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff3980682f0 0x7ff39806a7a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009870 tx=0x7ff3b0008040 comp rx=0 tx=0).stop 2023-11-08T08:20:29.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 msgr2=0x7ff3b4154ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b4154ee0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40b6eb0 tx=0x7ff3ac004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 shutdown_connections 2023-11-08T08:20:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3b415e010 0x7ff3b4154ee0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff3b406e400 0x7ff3b4148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff3980682f0 0x7ff39806a7a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 --2- 172.21.15.116:0/2005337790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3b40d15f0 0x7ff3b4151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 >> 172.21.15.116:0/2005337790 conn(0x7ff3b406a270 msgr2=0x7ff3b4149980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 shutdown_connections 2023-11-08T08:20:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.481+0000 7ff3bbe74700 1 -- 172.21.15.116:0/2005337790 wait complete. 2023-11-08T08:20:29.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:20:29.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:20:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.618+0000 7f31108db700 1 Processor -- start 2023-11-08T08:20:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.618+0000 7f31108db700 1 -- start start 2023-11-08T08:20:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f31108db700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f310c06a980 0x7f310c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f31108db700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c15cc10 con 0x7f310c0cf710 2023-11-08T08:20:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f31108db700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c06c9b0 con 0x7f310c071890 2023-11-08T08:20:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f31108db700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c06cc90 con 0x7f310c06a980 2023-11-08T08:20:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f3109d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f310c06a980 0x7f310c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f310a59c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f310c0cf710 0x7f310c06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39878/0 (socket says 172.21.15.116:39878) 2023-11-08T08:20:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f310a59c700 1 -- 172.21.15.116:0/3317690721 learned_addr learned my addr 172.21.15.116:0/3317690721 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f3109d9b700 1 -- 172.21.15.116:0/3317690721 >> v1:172.21.15.129:6789/0 conn(0x7f310c071890 legacy=0x7f310c160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f3109d9b700 1 -- 172.21.15.116:0/3317690721 >> v1:172.21.15.116:6789/0 conn(0x7f310c0cf710 legacy=0x7f310c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f3109d9b700 1 -- 172.21.15.116:0/3317690721 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c06cf70 con 0x7f310c06a980 2023-11-08T08:20:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f3109d9b700 1 --2- 172.21.15.116:0/3317690721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c06a980 0x7f310c0ce740 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3100009840 tx=0x7f3100009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145ee1d4f0fd9d1e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f310959a700 1 -- 172.21.15.116:0/3317690721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100004230 con 0x7f310c06a980 2023-11-08T08:20:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f310959a700 1 -- 172.21.15.116:0/3317690721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100004390 con 0x7f310c06a980 2023-11-08T08:20:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.619+0000 7f310959a700 1 -- 172.21.15.116:0/3317690721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000036a0 con 0x7f310c06a980 2023-11-08T08:20:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 -- 172.21.15.116:0/3317690721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c06a980 msgr2=0x7f310c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 --2- 172.21.15.116:0/3317690721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c06a980 0x7f310c0ce740 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3100009840 tx=0x7f3100009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 -- 172.21.15.116:0/3317690721 shutdown_connections 2023-11-08T08:20:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 --2- 172.21.15.116:0/3317690721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c06a980 0x7f310c0ce740 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 -- 172.21.15.116:0/3317690721 >> 172.21.15.116:0/3317690721 conn(0x7f310c06c010 msgr2=0x7f310c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 -- 172.21.15.116:0/3317690721 shutdown_connections 2023-11-08T08:20:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.620+0000 7f31108db700 1 -- 172.21.15.116:0/3317690721 wait complete. 2023-11-08T08:20:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.621+0000 7f31108db700 1 Processor -- start 2023-11-08T08:20:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.621+0000 7f31108db700 1 -- start start 2023-11-08T08:20:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f31108db700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f310c06a980 0x7f310c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f31108db700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f310c071890 0x7f310c151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f31108db700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 0x7f310c0b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f31108db700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0b5890 con 0x7f310c0cf710 2023-11-08T08:20:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f31108db700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0b42b0 con 0x7f310c071890 2023-11-08T08:20:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f31108db700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0b4590 con 0x7f310c06a980 2023-11-08T08:20:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 0x7f310c0b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 0x7f310c0b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39578/0 (socket says 172.21.15.116:39578) 2023-11-08T08:20:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 -- 172.21.15.116:0/1349959491 learned_addr learned my addr 172.21.15.116:0/1349959491 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310a59c700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f310c06a980 0x7f310c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f3109d9b700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f310c071890 0x7f310c151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 -- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f310c06a980 msgr2=0x7f310c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f310c06a980 0x7f310c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 -- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f310c071890 msgr2=0x7f310c151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f310c071890 0x7f310c151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.622+0000 7f310ad9d700 1 -- 172.21.15.116:0/1349959491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000094a0 con 0x7f310c0cf710 2023-11-08T08:20:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.623+0000 7f310ad9d700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 0x7f310c0b38d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f310400c7a0 tx=0x7f310400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.623+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104003e10 con 0x7f310c0cf710 2023-11-08T08:20:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.623+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104005070 con 0x7f310c0cf710 2023-11-08T08:20:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.623+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104005420 con 0x7f310c0cf710 2023-11-08T08:20:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.623+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c06d530 con 0x7f310c0cf710 2023-11-08T08:20:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.623+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c06d250 con 0x7f310c0cf710 2023-11-08T08:20:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.624+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c071620 con 0x7f310c0cf710 2023-11-08T08:20:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.624+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3104010070 con 0x7f310c0cf710 2023-11-08T08:20:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.625+0000 7f30faffd700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30ec0682f0 0x7f30ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.625+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3104084bf0 con 0x7f310c0cf710 2023-11-08T08:20:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.625+0000 7f310a59c700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30ec0682f0 0x7f30ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.625+0000 7f310a59c700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30ec0682f0 0x7f30ec06a7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000f60 tx=0x7f30fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.629+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3104051710 con 0x7f310c0cf710 2023-11-08T08:20:29.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.809+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f310c0743f0 con 0x7f310c0cf710 2023-11-08T08:20:29.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.809+0000 7f30faffd700 1 -- 172.21.15.116:0/1349959491 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v40) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3104017070 con 0x7f310c0cf710 2023-11-08T08:20:29.810 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:20:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30ec0682f0 msgr2=0x7f30ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30ec0682f0 0x7f30ec06a7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000f60 tx=0x7f30fc009040 comp rx=0 tx=0).stop 2023-11-08T08:20:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 msgr2=0x7f310c0b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 0x7f310c0b38d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f310400c7a0 tx=0x7f310400cb60 comp rx=0 tx=0).stop 2023-11-08T08:20:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 shutdown_connections 2023-11-08T08:20:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310c0cf710 0x7f310c0b38d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f310c06a980 0x7f310c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f30ec0682f0 0x7f30ec06a7a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 --2- 172.21.15.116:0/1349959491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f310c071890 0x7f310c151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 >> 172.21.15.116:0/1349959491 conn(0x7f310c06c010 msgr2=0x7f310c15c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 shutdown_connections 2023-11-08T08:20:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.811+0000 7f31108db700 1 -- 172.21.15.116:0/1349959491 wait complete. 2023-11-08T08:20:29.824 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:20:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.946+0000 7fe2a65c1700 1 Processor -- start 2023-11-08T08:20:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.946+0000 7fe2a65c1700 1 -- start start 2023-11-08T08:20:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a65c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a015a590 0x7fe2a006b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a65c1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe2a00b50d0 0x7fe2a006c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a65c1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a015a960 0x7fe2a006c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a65c1700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a00cd480 con 0x7fe2a015a960 2023-11-08T08:20:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a65c1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a00cd760 con 0x7fe2a00b50d0 2023-11-08T08:20:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a65c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00cda40 con 0x7fe2a015a590 2023-11-08T08:20:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a4b5e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a015a960 0x7fe2a006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a4b5e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a015a960 0x7fe2a006c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39590/0 (socket says 172.21.15.116:39590) 2023-11-08T08:20:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.947+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/4255768945 learned_addr learned my addr 172.21.15.116:0/4255768945 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe29ffff700 1 --2- 172.21.15.116:0/4255768945 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a015a590 0x7fe2a006b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe29f7fe700 1 --2- 172.21.15.116:0/4255768945 >> v2:172.21.15.129:3300/0 conn(0x7fe2a00b50d0 0x7fe2a006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/4255768945 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a015a590 msgr2=0x7fe2a006b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a4b5e700 1 --2- 172.21.15.116:0/4255768945 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a015a590 0x7fe2a006b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/4255768945 >> v2:172.21.15.129:3300/0 conn(0x7fe2a00b50d0 msgr2=0x7fe2a006c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:29.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a4b5e700 1 --2- 172.21.15.116:0/4255768945 >> v2:172.21.15.129:3300/0 conn(0x7fe2a00b50d0 0x7fe2a006c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/4255768945 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00cdd20 con 0x7fe2a015a960 2023-11-08T08:20:29.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a4b5e700 1 --2- 172.21.15.116:0/4255768945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a006c980 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00cbd70 tx=0x7fe29800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e32e32844f3c3d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe29effd700 1 -- 172.21.15.116:0/4255768945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29800e070 con 0x7fe2a015a960 2023-11-08T08:20:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe29effd700 1 -- 172.21.15.116:0/4255768945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29800b9c0 con 0x7fe2a015a960 2023-11-08T08:20:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe29effd700 1 -- 172.21.15.116:0/4255768945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29800bd70 con 0x7fe2a015a960 2023-11-08T08:20:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 -- 172.21.15.116:0/4255768945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 msgr2=0x7fe2a006c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/4255768945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a006c980 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00cbd70 tx=0x7fe29800b330 comp rx=0 tx=0).stop 2023-11-08T08:20:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 -- 172.21.15.116:0/4255768945 shutdown_connections 2023-11-08T08:20:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/4255768945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a006c980 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/4255768945 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a015a590 0x7fe2a006b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/4255768945 >> v2:172.21.15.129:3300/0 conn(0x7fe2a00b50d0 0x7fe2a006c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.948+0000 7fe2a65c1700 1 -- 172.21.15.116:0/4255768945 >> 172.21.15.116:0/4255768945 conn(0x7fe2a00cf400 msgr2=0x7fe2a00cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.949+0000 7fe2a65c1700 1 -- 172.21.15.116:0/4255768945 shutdown_connections 2023-11-08T08:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.949+0000 7fe2a65c1700 1 -- 172.21.15.116:0/4255768945 wait complete. 2023-11-08T08:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.950+0000 7fe2a65c1700 1 Processor -- start 2023-11-08T08:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.950+0000 7fe2a65c1700 1 -- start start 2023-11-08T08:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a65c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a00b50d0 0x7fe2a00c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a65c1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2a015a590 0x7fe2a00c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a65c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a00c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a65c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00c5330 con 0x7fe2a015a960 2023-11-08T08:20:29.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a65c1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00b2ea0 con 0x7fe2a015a590 2023-11-08T08:20:29.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a65c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00b3180 con 0x7fe2a00b50d0 2023-11-08T08:20:29.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a4b5e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a00c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a4b5e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a00c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39592/0 (socket says 172.21.15.116:39592) 2023-11-08T08:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/3218960072 learned_addr learned my addr 172.21.15.116:0/3218960072 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe29ffff700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a00b50d0 0x7fe2a00c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a00b50d0 msgr2=0x7fe2a00c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.951+0000 7fe2a4b5e700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a00b50d0 0x7fe2a00c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2a015a590 msgr2=0x7fe2a00c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe2a4b5e700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2a015a590 0x7fe2a00c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe2a4b5e700 1 -- 172.21.15.116:0/3218960072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe298009b30 con 0x7fe2a015a960 2023-11-08T08:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe2a4b5e700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a00c3470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe2a006fef0 tx=0x7fe298004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2980099b0 con 0x7fe2a015a960 2023-11-08T08:20:29.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2980092e0 con 0x7fe2a015a960 2023-11-08T08:20:29.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29801b630 con 0x7fe2a015a960 2023-11-08T08:20:29.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a006baf0 con 0x7fe2a015a960 2023-11-08T08:20:29.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.952+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a00ce000 con 0x7fe2a015a960 2023-11-08T08:20:29.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.953+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a015e280 con 0x7fe2a015a960 2023-11-08T08:20:29.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.953+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe298021030 con 0x7fe2a015a960 2023-11-08T08:20:29.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.953+0000 7fe29cff9700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe28c0682f0 0x7fe28c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:29.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.954+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe298086f00 con 0x7fe2a015a960 2023-11-08T08:20:29.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.954+0000 7fe29ffff700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe28c0682f0 0x7fe28c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:29.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.954+0000 7fe29ffff700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe28c0682f0 0x7fe28c06a7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe29400d8d0 tx=0x7fe294008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:29.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:29.958+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2980539c0 con 0x7fe2a015a960 2023-11-08T08:20:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.137+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2a00743f0 con 0x7fe2a015a960 2023-11-08T08:20:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.137+0000 7fe29cff9700 1 -- 172.21.15.116:0/3218960072 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe298056fe0 con 0x7fe2a015a960 2023-11-08T08:20:30.139 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:20:30.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe28c0682f0 msgr2=0x7fe28c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe28c0682f0 0x7fe28c06a7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe29400d8d0 tx=0x7fe294008040 comp rx=0 tx=0).stop 2023-11-08T08:20:30.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 msgr2=0x7fe2a00c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a00c3470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe2a006fef0 tx=0x7fe298004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 shutdown_connections 2023-11-08T08:20:30.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a015a960 0x7fe2a00c3470 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe2a00b50d0 0x7fe2a00c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe28c0682f0 0x7fe28c06a7a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 --2- 172.21.15.116:0/3218960072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2a015a590 0x7fe2a00c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.139+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 >> 172.21.15.116:0/3218960072 conn(0x7fe2a00cf400 msgr2=0x7fe2a0072070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.140+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 shutdown_connections 2023-11-08T08:20:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.140+0000 7fe2a65c1700 1 -- 172.21.15.116:0/3218960072 wait complete. 2023-11-08T08:20:30.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-08T08:20:30.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.271+0000 7fc080180700 1 Processor -- start 2023-11-08T08:20:30.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.272+0000 7fc080180700 1 -- start start 2023-11-08T08:20:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc080180700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc0780ce040 0x7fc07815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc080180700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07813fc90 con 0x7fc078142db0 2023-11-08T08:20:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc080180700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07813f480 con 0x7fc0780b1770 2023-11-08T08:20:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc080180700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07813f6d0 con 0x7fc0780ce040 2023-11-08T08:20:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07d71b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc078142db0 0x7fc07813ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39880/0 (socket says 172.21.15.116:39880) 2023-11-08T08:20:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07d71b700 1 -- 172.21.15.116:0/2916828159 learned_addr learned my addr 172.21.15.116:0/2916828159 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07df1c700 1 --2- 172.21.15.116:0/2916828159 >> v2:172.21.15.129:3300/0 conn(0x7fc0780ce040 0x7fc07815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979646264 0 0) 0x7fc07813fc90 con 0x7fc078142db0 2023-11-08T08:20:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0640035b0 con 0x7fc078142db0 2023-11-08T08:20:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3370658634 0 0) 0x7fc0640035b0 con 0x7fc078142db0 2023-11-08T08:20:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 >> v2:172.21.15.129:3300/0 conn(0x7fc0780ce040 msgr2=0x7fc07815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 --2- 172.21.15.116:0/2916828159 >> v2:172.21.15.129:3300/0 conn(0x7fc0780ce040 0x7fc07815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 >> v1:172.21.15.129:6789/0 conn(0x7fc0780b1770 legacy=0x7fc0780b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07813f9b0 con 0x7fc078142db0 2023-11-08T08:20:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fc074001670 con 0x7fc078142db0 2023-11-08T08:20:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 >> v1:172.21.15.116:6789/0 conn(0x7fc078142db0 legacy=0x7fc07813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064003a00 0x7fc064003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc064004850 0x7fc064004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc064005200 0x7fc06400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07813f6d0 con 0x7fc064005200 2023-11-08T08:20:30.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06400c080 con 0x7fc064004850 2023-11-08T08:20:30.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.273+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06400e450 con 0x7fc064003a00 2023-11-08T08:20:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc064005200 0x7fc06400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07d71b700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc064004850 0x7fc064004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07df1c700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064003a00 0x7fc064003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 -- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064003a00 msgr2=0x7fc064003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064003a00 0x7fc064003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 -- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc064004850 msgr2=0x7fc064004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc064004850 0x7fc064004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 -- 172.21.15.116:0/2916828159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0640035b0 con 0x7fc064005200 2023-11-08T08:20:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07df1c700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064003a00 0x7fc064003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 -- 172.21.15.116:0/2916828159 reap_dead start 2023-11-08T08:20:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc064005200 0x7fc06400ba10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc07000c190 tx=0x7fc064005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199b134341af77ee server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070001db0 con 0x7fc064005200 2023-11-08T08:20:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07000ccf0 con 0x7fc064005200 2023-11-08T08:20:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.274+0000 7fc07cf1a700 1 -- 172.21.15.116:0/2916828159 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070003c40 con 0x7fc064005200 2023-11-08T08:20:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 -- 172.21.15.116:0/2916828159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc064005200 msgr2=0x7fc06400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc064005200 0x7fc06400ba10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc07000c190 tx=0x7fc064005880 comp rx=0 tx=0).stop 2023-11-08T08:20:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 -- 172.21.15.116:0/2916828159 shutdown_connections 2023-11-08T08:20:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 --2- 172.21.15.116:0/2916828159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc064005200 0x7fc06400ba10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 -- 172.21.15.116:0/2916828159 >> 172.21.15.116:0/2916828159 conn(0x7fc07806c010 msgr2=0x7fc0780cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 -- 172.21.15.116:0/2916828159 shutdown_connections 2023-11-08T08:20:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.275+0000 7fc080180700 1 -- 172.21.15.116:0/2916828159 wait complete. 2023-11-08T08:20:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.276+0000 7fc080180700 1 Processor -- start 2023-11-08T08:20:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.276+0000 7fc080180700 1 -- start start 2023-11-08T08:20:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064005200 0x7fc0780b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0780b5730 0x7fc078140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 0x7fc0781f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0781418c0 con 0x7fc078141450 2023-11-08T08:20:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780b3900 con 0x7fc0780b5730 2023-11-08T08:20:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780b3280 con 0x7fc064005200 2023-11-08T08:20:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 0x7fc0781f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 0x7fc0781f5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39612/0 (socket says 172.21.15.116:39612) 2023-11-08T08:20:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 -- 172.21.15.116:0/2770825345 learned_addr learned my addr 172.21.15.116:0/2770825345 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07df1c700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064005200 0x7fc0780b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07d71b700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0780b5730 0x7fc078140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 -- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064005200 msgr2=0x7fc0780b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064005200 0x7fc0780b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 -- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0780b5730 msgr2=0x7fc078140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0780b5730 0x7fc078140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 -- 172.21.15.116:0/2770825345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07815d3e0 con 0x7fc078141450 2023-11-08T08:20:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc07e71d700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 0x7fc0781f5b30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc070005040 tx=0x7fc0700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070005270 con 0x7fc078141450 2023-11-08T08:20:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0700053d0 con 0x7fc078141450 2023-11-08T08:20:30.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07815d100 con 0x7fc078141450 2023-11-08T08:20:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070005740 con 0x7fc078141450 2023-11-08T08:20:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.277+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0780ce410 con 0x7fc078141450 2023-11-08T08:20:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.278+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0780743f0 con 0x7fc078141450 2023-11-08T08:20:30.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.279+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc070026030 con 0x7fc078141450 2023-11-08T08:20:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.279+0000 7fc06e7fc700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc05c0682f0 0x7fc05c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.279+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc070086370 con 0x7fc078141450 2023-11-08T08:20:30.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.279+0000 7fc07df1c700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc05c0682f0 0x7fc05c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.280+0000 7fc07df1c700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc05c0682f0 0x7fc05c06a7a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc068009b40 tx=0x7fc068008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:30.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.283+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc070052f70 con 0x7fc078141450 2023-11-08T08:20:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.471+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc078157da0 con 0x7fc078141450 2023-11-08T08:20:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.471+0000 7fc06e7fc700 1 -- 172.21.15.116:0/2770825345 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) v1 ==== 84+0+126 (secure 0 0 0) 0x7fc070016090 con 0x7fc078141450 2023-11-08T08:20:30.473 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:30.474 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-08T08:20:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-08T08:20:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": "rbd" 2023-11-08T08:20:30.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 2, 2023-11-08T08:20:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-08T08:20:30.475 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:30.475 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-08T08:20:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc05c0682f0 msgr2=0x7fc05c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc05c0682f0 0x7fc05c06a7a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc068009b40 tx=0x7fc068008040 comp rx=0 tx=0).stop 2023-11-08T08:20:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 msgr2=0x7fc0781f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 0x7fc0781f5b30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc070005040 tx=0x7fc0700014a0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 shutdown_connections 2023-11-08T08:20:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc078141450 0x7fc0781f5b30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc064005200 0x7fc0780b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc05c0682f0 0x7fc05c06a7a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 --2- 172.21.15.116:0/2770825345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0780b5730 0x7fc078140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 >> 172.21.15.116:0/2770825345 conn(0x7fc07806c010 msgr2=0x7fc078157000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 shutdown_connections 2023-11-08T08:20:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.474+0000 7fc080180700 1 -- 172.21.15.116:0/2770825345 wait complete. 2023-11-08T08:20:30.487 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-08T08:20:30.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-08T08:20:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.611+0000 7f8b9d451700 1 Processor -- start 2023-11-08T08:20:30.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.611+0000 7f8b9d451700 1 -- start start 2023-11-08T08:20:30.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b9d451700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8b9806b9f0 0x7f8b9815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b9d451700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980cdc10 con 0x7f8b9815cce0 2023-11-08T08:20:30.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b9d451700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9806be50 con 0x7f8b98071320 2023-11-08T08:20:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b9d451700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9815d0b0 con 0x7f8b9806b9f0 2023-11-08T08:20:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b96ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8b9815cce0 0x7f8b98070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39886/0 (socket says 172.21.15.116:39886) 2023-11-08T08:20:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b96ffd700 1 -- 172.21.15.116:0/3982814673 learned_addr learned my addr 172.21.15.116:0/3982814673 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b977fe700 1 --2- 172.21.15.116:0/3982814673 >> v2:172.21.15.129:3300/0 conn(0x7f8b9806b9f0 0x7f8b9815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b977fe700 1 -- 172.21.15.116:0/3982814673 >> v1:172.21.15.129:6789/0 conn(0x7f8b98071320 legacy=0x7f8b98160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b977fe700 1 -- 172.21.15.116:0/3982814673 >> v1:172.21.15.116:6789/0 conn(0x7f8b9815cce0 legacy=0x7f8b98070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.612+0000 7f8b977fe700 1 -- 172.21.15.116:0/3982814673 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9815e180 con 0x7f8b9806b9f0 2023-11-08T08:20:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.613+0000 7f8b977fe700 1 --2- 172.21.15.116:0/3982814673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b9806b9f0 0x7f8b9815dc40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009d90 tx=0x7f8b9000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1446f14429f7e01b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.613+0000 7f8b95ffb700 1 -- 172.21.15.116:0/3982814673 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000e070 con 0x7f8b9806b9f0 2023-11-08T08:20:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.613+0000 7f8b95ffb700 1 -- 172.21.15.116:0/3982814673 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9000bbd0 con 0x7f8b9806b9f0 2023-11-08T08:20:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.613+0000 7f8b95ffb700 1 -- 172.21.15.116:0/3982814673 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90012430 con 0x7f8b9806b9f0 2023-11-08T08:20:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 -- 172.21.15.116:0/3982814673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b9806b9f0 msgr2=0x7f8b9815dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 --2- 172.21.15.116:0/3982814673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b9806b9f0 0x7f8b9815dc40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009d90 tx=0x7f8b9000b330 comp rx=0 tx=0).stop 2023-11-08T08:20:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 -- 172.21.15.116:0/3982814673 shutdown_connections 2023-11-08T08:20:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 --2- 172.21.15.116:0/3982814673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b9806b9f0 0x7f8b9815dc40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 -- 172.21.15.116:0/3982814673 >> 172.21.15.116:0/3982814673 conn(0x7f8b980cfc30 msgr2=0x7f8b980d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 -- 172.21.15.116:0/3982814673 shutdown_connections 2023-11-08T08:20:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.614+0000 7f8b9d451700 1 -- 172.21.15.116:0/3982814673 wait complete. 2023-11-08T08:20:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.615+0000 7f8b9d451700 1 Processor -- start 2023-11-08T08:20:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.615+0000 7f8b9d451700 1 -- start start 2023-11-08T08:20:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 0x7f8b980b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b98071320 0x7f8b98153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b9815cce0 0x7f8b98156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98154b40 con 0x7f8b9806b9f0 2023-11-08T08:20:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980b3df0 con 0x7f8b98071320 2023-11-08T08:20:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98153e60 con 0x7f8b9815cce0 2023-11-08T08:20:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 0x7f8b980b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 0x7f8b980b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39624/0 (socket says 172.21.15.116:39624) 2023-11-08T08:20:30.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 -- 172.21.15.116:0/1285102554 learned_addr learned my addr 172.21.15.116:0/1285102554 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:30.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b967fc700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b98071320 0x7f8b98153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b977fe700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b9815cce0 0x7f8b98156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 -- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b9815cce0 msgr2=0x7f8b98156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b9815cce0 0x7f8b98156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 -- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b98071320 msgr2=0x7f8b98153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b98071320 0x7f8b98153920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 -- 172.21.15.116:0/1285102554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b90009a70 con 0x7f8b9806b9f0 2023-11-08T08:20:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b967fc700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b98071320 0x7f8b98153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b96ffd700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 0x7f8b980b38b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800c480 tx=0x7f8b8800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88005d70 con 0x7f8b9806b9f0 2023-11-08T08:20:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b880070d0 con 0x7f8b9806b9f0 2023-11-08T08:20:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b980cf730 con 0x7f8b9806b9f0 2023-11-08T08:20:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88007440 con 0x7f8b9806b9f0 2023-11-08T08:20:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.616+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b980cfa10 con 0x7f8b9806b9f0 2023-11-08T08:20:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.617+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b980cd8a0 con 0x7f8b9806b9f0 2023-11-08T08:20:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.618+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b88019030 con 0x7f8b9806b9f0 2023-11-08T08:20:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.618+0000 7f8b7f7fe700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b84068320 0x7f8b8406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.618+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8b88084c60 con 0x7f8b9806b9f0 2023-11-08T08:20:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.618+0000 7f8b967fc700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b84068320 0x7f8b8406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.619+0000 7f8b967fc700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b84068320 0x7f8b8406a7d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b981538d0 tx=0x7f8b8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.622+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b8804b5e0 con 0x7f8b9806b9f0 2023-11-08T08:20:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:30.803+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b9806d910 con 0x7f8b9806b9f0 2023-11-08T08:20:31.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.624+0000 7f8b7f7fe700 1 -- 172.21.15.116:0/1285102554 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8b880518c0 con 0x7f8b9806b9f0 2023-11-08T08:20:31.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.628+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b84068320 msgr2=0x7f8b8406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.628+0000 7f8b9d451700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b84068320 0x7f8b8406a7d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b981538d0 tx=0x7f8b8c009040 comp rx=0 tx=0).stop 2023-11-08T08:20:31.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.628+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 msgr2=0x7f8b980b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.628+0000 7f8b9d451700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 0x7f8b980b38b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800c480 tx=0x7f8b8800c840 comp rx=0 tx=0).stop 2023-11-08T08:20:31.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 shutdown_connections 2023-11-08T08:20:31.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8b9815cce0 0x7f8b98156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9806b9f0 0x7f8b980b38b0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8b84068320 0x7f8b8406a7d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 --2- 172.21.15.116:0/1285102554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b98071320 0x7f8b98153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 >> 172.21.15.116:0/1285102554 conn(0x7f8b980cfc30 msgr2=0x7f8b980d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:31.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 shutdown_connections 2023-11-08T08:20:31.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.629+0000 7f8b9d451700 1 -- 172.21.15.116:0/1285102554 wait complete. 2023-11-08T08:20:31.632 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-08T08:20:31.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-08T08:20:31.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.777+0000 7f37fe075700 1 Processor -- start 2023-11-08T08:20:31.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.777+0000 7f37fe075700 1 -- start start 2023-11-08T08:20:31.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37fe075700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37f80af0a0 0x7f37f815a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:31.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37fe075700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f8159ec0 con 0x7f37f80b1650 2023-11-08T08:20:31.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37fe075700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f815d8a0 con 0x7f37f81599b0 2023-11-08T08:20:31.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37fe075700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f815db80 con 0x7f37f80af0a0 2023-11-08T08:20:31.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f7fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37f80af0a0 0x7f37f815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:31.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f77fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f37f80b1650 0x7f37f80b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39896/0 (socket says 172.21.15.116:39896) 2023-11-08T08:20:31.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f7fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37f80af0a0 0x7f37f815a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39630/0 (socket says 172.21.15.116:39630) 2023-11-08T08:20:31.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f77fe700 1 -- 172.21.15.116:0/2406248905 learned_addr learned my addr 172.21.15.116:0/2406248905 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:31.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f7fff700 1 -- 172.21.15.116:0/2406248905 >> v1:172.21.15.129:6789/0 conn(0x7f37f81599b0 legacy=0x7f37f815d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f7fff700 1 -- 172.21.15.116:0/2406248905 >> v1:172.21.15.116:6789/0 conn(0x7f37f80b1650 legacy=0x7f37f80b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.778+0000 7f37f7fff700 1 -- 172.21.15.116:0/2406248905 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f815de60 con 0x7f37f80af0a0 2023-11-08T08:20:31.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37f7fff700 1 --2- 172.21.15.116:0/2406248905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80af0a0 0x7f37f815a350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009d90 tx=0x7f37f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed526b08c69c0f86 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:31.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37f67fc700 1 -- 172.21.15.116:0/2406248905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f000bdd0 con 0x7f37f80af0a0 2023-11-08T08:20:31.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37f67fc700 1 -- 172.21.15.116:0/2406248905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f0011460 con 0x7f37f80af0a0 2023-11-08T08:20:31.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37f67fc700 1 -- 172.21.15.116:0/2406248905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0011810 con 0x7f37f80af0a0 2023-11-08T08:20:31.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 -- 172.21.15.116:0/2406248905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80af0a0 msgr2=0x7f37f815a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 --2- 172.21.15.116:0/2406248905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80af0a0 0x7f37f815a350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009d90 tx=0x7f37f000b330 comp rx=0 tx=0).stop 2023-11-08T08:20:31.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 -- 172.21.15.116:0/2406248905 shutdown_connections 2023-11-08T08:20:31.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 --2- 172.21.15.116:0/2406248905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80af0a0 0x7f37f815a350 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 -- 172.21.15.116:0/2406248905 >> 172.21.15.116:0/2406248905 conn(0x7f37f806c010 msgr2=0x7f37f80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:31.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 -- 172.21.15.116:0/2406248905 shutdown_connections 2023-11-08T08:20:31.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.779+0000 7f37fe075700 1 -- 172.21.15.116:0/2406248905 wait complete. 2023-11-08T08:20:31.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.781+0000 7f37fe075700 1 Processor -- start 2023-11-08T08:20:31.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.781+0000 7f37fe075700 1 -- start start 2023-11-08T08:20:31.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37fe075700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f80af0a0 0x7f37f80c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:31.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37fe075700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37f80b1650 0x7f37f81f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:31.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37fe075700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 0x7f37f80c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:31.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37fe075700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80c7fb0 con 0x7f37f80c5070 2023-11-08T08:20:31.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37fe075700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80c8290 con 0x7f37f80af0a0 2023-11-08T08:20:31.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37fe075700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80c5b10 con 0x7f37f80b1650 2023-11-08T08:20:31.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 0x7f37f80c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:31.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 0x7f37f80c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39634/0 (socket says 172.21.15.116:39634) 2023-11-08T08:20:31.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 -- 172.21.15.116:0/3361877906 learned_addr learned my addr 172.21.15.116:0/3361877906 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:31.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f6ffd700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37f80b1650 0x7f37f81f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:31.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f77fe700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f80af0a0 0x7f37f80c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:31.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 -- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37f80b1650 msgr2=0x7f37f81f1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37f80b1650 0x7f37f81f1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 -- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f80af0a0 msgr2=0x7f37f80c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:31.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f80af0a0 0x7f37f80c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:31.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f7fff700 1 -- 172.21.15.116:0/3361877906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f0009a70 con 0x7f37f80c5070 2023-11-08T08:20:31.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f77fe700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f80af0a0 0x7f37f80c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:31.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.782+0000 7f37f6ffd700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37f80b1650 0x7f37f81f1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:31.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.783+0000 7f37f7fff700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 0x7f37f80c6310 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f37f80708b0 tx=0x7f37f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:31.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.783+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0020070 con 0x7f37f80c5070 2023-11-08T08:20:31.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.783+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f0011990 con 0x7f37f80c5070 2023-11-08T08:20:31.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.783+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f815aa20 con 0x7f37f80c5070 2023-11-08T08:20:31.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.783+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0011d40 con 0x7f37f80c5070 2023-11-08T08:20:31.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.783+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f815a890 con 0x7f37f80c5070 2023-11-08T08:20:31.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.784+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f80743f0 con 0x7f37f80c5070 2023-11-08T08:20:31.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.784+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37f002b030 con 0x7f37f80c5070 2023-11-08T08:20:31.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.784+0000 7f37dffff700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37e40682f0 0x7f37e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:31.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.784+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f37f0089460 con 0x7f37f80c5070 2023-11-08T08:20:31.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.784+0000 7f37f77fe700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37e40682f0 0x7f37e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:31.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.785+0000 7f37f77fe700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37e40682f0 0x7f37e406a7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f37ec005910 tx=0x7f37ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:31.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.789+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37f0055db0 con 0x7f37f80c5070 2023-11-08T08:20:31.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:31.970+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f37f806cd10 con 0x7f37f80c5070 2023-11-08T08:20:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.628+0000 7f37dffff700 1 -- 172.21.15.116:0/3361877906 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) v1 ==== 170+0+0 (secure 0 0 0) 0x7f37f00593d0 con 0x7f37f80c5070 2023-11-08T08:20:32.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.635+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37e40682f0 msgr2=0x7f37e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:32.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.635+0000 7f37fe075700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37e40682f0 0x7f37e406a7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f37ec005910 tx=0x7f37ec009040 comp rx=0 tx=0).stop 2023-11-08T08:20:32.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.635+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 msgr2=0x7f37f80c6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:32.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.635+0000 7f37fe075700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 0x7f37f80c6310 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f37f80708b0 tx=0x7f37f0004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.636+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 shutdown_connections 2023-11-08T08:20:32.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.636+0000 7f37fe075700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37f80c5070 0x7f37f80c6310 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.636+0000 7f37fe075700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f37e40682f0 0x7f37e406a7a0 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f37ec005910 tx=0x7f37ec009040 comp rx=0 tx=0).stop 2023-11-08T08:20:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.636+0000 7f37fe075700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f80af0a0 0x7f37f80c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.636+0000 7f37fe075700 1 --2- 172.21.15.116:0/3361877906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f37f80b1650 0x7f37f81f1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.636+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 >> 172.21.15.116:0/3361877906 conn(0x7f37f806c010 msgr2=0x7f37f81590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.637+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 shutdown_connections 2023-11-08T08:20:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.638+0000 7f37fe075700 1 -- 172.21.15.116:0/3361877906 wait complete. 2023-11-08T08:20:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-08T08:20:32.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-08T08:20:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.779+0000 7fd5a6469700 1 Processor -- start 2023-11-08T08:20:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.779+0000 7fd5a6469700 1 -- start start 2023-11-08T08:20:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd5a6469700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd5a015c810 0x7fd5a0160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd5a6469700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a006a270 0x7fd5a006a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd5a6469700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a006abf0 con 0x7fd5a00718e0 2023-11-08T08:20:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd5a6469700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a015cbe0 con 0x7fd5a015c810 2023-11-08T08:20:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd5a6469700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a01607a0 con 0x7fd5a006a270 2023-11-08T08:20:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd59ffff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd5a015c810 0x7fd5a0160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd5a4a06700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a006a270 0x7fd5a006a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd59f7fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd5a00718e0 0x7fd5a006c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:39728/0 (socket says 172.21.15.116:39728) 2023-11-08T08:20:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.780+0000 7fd59f7fe700 1 -- 172.21.15.116:0/4052916858 learned_addr learned my addr 172.21.15.116:0/4052916858 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a4a06700 1 -- 172.21.15.116:0/4052916858 >> v2:172.21.15.129:3300/0 conn(0x7fd5a015c810 msgr2=0x7fd5a0160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a4a06700 1 --2- 172.21.15.116:0/4052916858 >> v2:172.21.15.129:3300/0 conn(0x7fd5a015c810 0x7fd5a0160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a4a06700 1 -- 172.21.15.116:0/4052916858 >> v1:172.21.15.129:6789/0 conn(0x7fd5a00718e0 legacy=0x7fd5a006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a4a06700 1 -- 172.21.15.116:0/4052916858 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a0160a50 con 0x7fd5a006a270 2023-11-08T08:20:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd59ffff700 1 --2- 172.21.15.116:0/4052916858 >> v2:172.21.15.129:3300/0 conn(0x7fd5a015c810 0x7fd5a0160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a4a06700 1 --2- 172.21.15.116:0/4052916858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a006a270 0x7fd5a006a6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49bc89dd4b0d431d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd59effd700 1 -- 172.21.15.116:0/4052916858 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59800e070 con 0x7fd5a006a270 2023-11-08T08:20:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd59effd700 1 -- 172.21.15.116:0/4052916858 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59800bb20 con 0x7fd5a006a270 2023-11-08T08:20:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd59effd700 1 -- 172.21.15.116:0/4052916858 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598012430 con 0x7fd5a006a270 2023-11-08T08:20:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a6469700 1 -- 172.21.15.116:0/4052916858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a006a270 msgr2=0x7fd5a006a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.781+0000 7fd5a6469700 1 --2- 172.21.15.116:0/4052916858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a006a270 0x7fd5a006a6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59800b330 comp rx=0 tx=0).stop 2023-11-08T08:20:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.782+0000 7fd5a6469700 1 -- 172.21.15.116:0/4052916858 shutdown_connections 2023-11-08T08:20:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.782+0000 7fd5a6469700 1 --2- 172.21.15.116:0/4052916858 >> v2:172.21.15.129:3300/0 conn(0x7fd5a015c810 0x7fd5a0160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.782+0000 7fd5a6469700 1 --2- 172.21.15.116:0/4052916858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a006a270 0x7fd5a006a6b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.782+0000 7fd5a6469700 1 -- 172.21.15.116:0/4052916858 >> 172.21.15.116:0/4052916858 conn(0x7fd5a006c010 msgr2=0x7fd5a00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.782+0000 7fd5a6469700 1 -- 172.21.15.116:0/4052916858 shutdown_connections 2023-11-08T08:20:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.782+0000 7fd5a6469700 1 -- 172.21.15.116:0/4052916858 wait complete. 2023-11-08T08:20:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.783+0000 7fd5a6469700 1 Processor -- start 2023-11-08T08:20:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.783+0000 7fd5a6469700 1 -- start start 2023-11-08T08:20:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 0x7fd5a0153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a00718e0 0x7fd5a00c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5a015c810 0x7fd5a00b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0153990 con 0x7fd5a006a270 2023-11-08T08:20:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00c4610 con 0x7fd5a015c810 2023-11-08T08:20:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00c48f0 con 0x7fd5a00718e0 2023-11-08T08:20:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 0x7fd5a0153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 0x7fd5a0153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41840/0 (socket says 172.21.15.116:41840) 2023-11-08T08:20:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 -- 172.21.15.116:0/2162759112 learned_addr learned my addr 172.21.15.116:0/2162759112 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59f7fe700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a00718e0 0x7fd5a00c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a4a06700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5a015c810 0x7fd5a00b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 -- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a00718e0 msgr2=0x7fd5a00c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a00718e0 0x7fd5a00c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 -- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5a015c810 msgr2=0x7fd5a00b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5a015c810 0x7fd5a00b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 -- 172.21.15.116:0/2162759112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940094a0 con 0x7fd5a006a270 2023-11-08T08:20:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59ffff700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 0x7fd5a0153450 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd59400ebe0 tx=0x7fd59400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59400c850 con 0x7fd5a006a270 2023-11-08T08:20:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59400c9b0 con 0x7fd5a006a270 2023-11-08T08:20:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd598009a70 con 0x7fd5a006a270 2023-11-08T08:20:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59400cd40 con 0x7fd5a006a270 2023-11-08T08:20:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.784+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a0161010 con 0x7fd5a006a270 2023-11-08T08:20:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.785+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a0160d30 con 0x7fd5a006a270 2023-11-08T08:20:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.786+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd594019030 con 0x7fd5a006a270 2023-11-08T08:20:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.786+0000 7fd59cff9700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd58c0682f0 0x7fd58c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.786+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fd594085b90 con 0x7fd5a006a270 2023-11-08T08:20:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.786+0000 7fd59f7fe700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd58c0682f0 0x7fd58c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.786+0000 7fd59f7fe700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd58c0682f0 0x7fd58c06a7a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd5a006d940 tx=0x7fd590007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:32.791+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd594052450 con 0x7fd5a006a270 2023-11-08T08:20:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.010+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd5a00af7a0 con 0x7fd5a006a270 2023-11-08T08:20:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.015+0000 7fd59cff9700 1 -- 172.21.15.116:0/2162759112 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd59400cea0 con 0x7fd5a006a270 2023-11-08T08:20:33.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd58c0682f0 msgr2=0x7fd58c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd58c0682f0 0x7fd58c06a7a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd5a006d940 tx=0x7fd590007f10 comp rx=0 tx=0).stop 2023-11-08T08:20:33.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 msgr2=0x7fd5a0153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 0x7fd5a0153450 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd59400ebe0 tx=0x7fd59400eef0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 shutdown_connections 2023-11-08T08:20:33.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd58c0682f0 0x7fd58c06a7a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5a015c810 0x7fd5a00b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5a006a270 0x7fd5a0153450 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 --2- 172.21.15.116:0/2162759112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd5a00718e0 0x7fd5a00c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.018+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 >> 172.21.15.116:0/2162759112 conn(0x7fd5a006c010 msgr2=0x7fd5a00adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:33.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.019+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 shutdown_connections 2023-11-08T08:20:33.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.019+0000 7fd5a6469700 1 -- 172.21.15.116:0/2162759112 wait complete. 2023-11-08T08:20:33.022 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 7 and data pool 8 2023-11-08T08:20:33.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.159+0000 7fda33378700 1 Processor -- start 2023-11-08T08:20:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.159+0000 7fda33378700 1 -- start start 2023-11-08T08:20:33.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.160+0000 7fda33378700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fda2c0b4660 0x7fda2c071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.160+0000 7fda33378700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.160+0000 7fda33378700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0cfb00 con 0x7fda2c0cff90 2023-11-08T08:20:33.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.160+0000 7fda33378700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c15aa20 con 0x7fda2c0b4660 2023-11-08T08:20:33.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.160+0000 7fda33378700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0cf4d0 con 0x7fda2c0b4a30 2023-11-08T08:20:33.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31114700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fda2c0b4660 0x7fda2c071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31915700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda30913700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fda2c0cff90 0x7fda2c06b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:39732/0 (socket says 172.21.15.116:39732) 2023-11-08T08:20:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda30913700 1 -- 172.21.15.116:0/1114660380 learned_addr learned my addr 172.21.15.116:0/1114660380 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31114700 1 -- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 msgr2=0x7fda2c0b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31114700 1 --2- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31114700 1 -- 172.21.15.116:0/1114660380 >> v1:172.21.15.129:6789/0 conn(0x7fda2c0cff90 legacy=0x7fda2c06b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31114700 1 -- 172.21.15.116:0/1114660380 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c072550 con 0x7fda2c0b4660 2023-11-08T08:20:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31915700 1 --2- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda31114700 1 --2- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c071f70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fda28009840 tx=0x7fda28009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f117c06ed85a3686 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda23fff700 1 -- 172.21.15.116:0/1114660380 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28011830 con 0x7fda2c0b4660 2023-11-08T08:20:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda23fff700 1 -- 172.21.15.116:0/1114660380 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28011990 con 0x7fda2c0b4660 2023-11-08T08:20:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.161+0000 7fda23fff700 1 -- 172.21.15.116:0/1114660380 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28011d20 con 0x7fda2c0b4660 2023-11-08T08:20:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 -- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 msgr2=0x7fda2c071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 --2- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c071f70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fda28009840 tx=0x7fda28009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 -- 172.21.15.116:0/1114660380 shutdown_connections 2023-11-08T08:20:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 --2- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c071f70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 --2- 172.21.15.116:0/1114660380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 -- 172.21.15.116:0/1114660380 >> 172.21.15.116:0/1114660380 conn(0x7fda2c0b2fc0 msgr2=0x7fda2c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 -- 172.21.15.116:0/1114660380 shutdown_connections 2023-11-08T08:20:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.162+0000 7fda33378700 1 -- 172.21.15.116:0/1114660380 wait complete. 2023-11-08T08:20:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.163+0000 7fda33378700 1 Processor -- start 2023-11-08T08:20:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.163+0000 7fda33378700 1 -- start start 2023-11-08T08:20:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda33378700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda33378700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda33378700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 0x7fda2c0ad200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda33378700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0c8b00 con 0x7fda2c0cff90 2023-11-08T08:20:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda33378700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0c7f80 con 0x7fda2c0b4660 2023-11-08T08:20:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda33378700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0c81d0 con 0x7fda2c0b4a30 2023-11-08T08:20:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 0x7fda2c0ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 0x7fda2c0ad200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41846/0 (socket says 172.21.15.116:41846) 2023-11-08T08:20:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 -- 172.21.15.116:0/779948287 learned_addr learned my addr 172.21.15.116:0/779948287 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda30913700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 -- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 msgr2=0x7fda2c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 -- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 msgr2=0x7fda2c155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31114700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.164+0000 7fda31915700 1 -- 172.21.15.116:0/779948287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24009a70 con 0x7fda2c0cff90 2023-11-08T08:20:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.165+0000 7fda31915700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 0x7fda2c0ad200 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fda2400efc0 tx=0x7fda2400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.165+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24009d70 con 0x7fda2c0cff90 2023-11-08T08:20:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.165+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2400f040 con 0x7fda2c0cff90 2023-11-08T08:20:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.165+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda280094a0 con 0x7fda2c0cff90 2023-11-08T08:20:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.165+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2400ce00 con 0x7fda2c0cff90 2023-11-08T08:20:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.165+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c15f170 con 0x7fda2c0cff90 2023-11-08T08:20:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.166+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c0cf790 con 0x7fda2c0cff90 2023-11-08T08:20:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.166+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda24014480 con 0x7fda2c0cff90 2023-11-08T08:20:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.167+0000 7fda21ffb700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda100682f0 0x7fda1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.167+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fda24086660 con 0x7fda2c0cff90 2023-11-08T08:20:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.167+0000 7fda31114700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda100682f0 0x7fda1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.167+0000 7fda31114700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda100682f0 0x7fda1006a7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fda280090b0 tx=0x7fda28008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.171+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda24052de0 con 0x7fda2c0cff90 2023-11-08T08:20:33.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.351+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda2c15f300 con 0x7fda2c0cff90 2023-11-08T08:20:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.352+0000 7fda21ffb700 1 -- 172.21.15.116:0/779948287 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17922 (secure 0 0 0) 0x7fda24056400 con 0x7fda2c0cff90 2023-11-08T08:20:33.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:33.354 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":43,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:33.012466+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda100682f0 msgr2=0x7fda1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda100682f0 0x7fda1006a7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fda280090b0 tx=0x7fda28008030 comp rx=0 tx=0).stop 2023-11-08T08:20:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 msgr2=0x7fda2c0ad200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 0x7fda2c0ad200 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fda2400efc0 tx=0x7fda2400c3a0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 shutdown_connections 2023-11-08T08:20:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda2c0cff90 0x7fda2c0ad200 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fda100682f0 0x7fda1006a7a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c0b4660 0x7fda2c155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 --2- 172.21.15.116:0/779948287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fda2c0b4a30 0x7fda2c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 >> 172.21.15.116:0/779948287 conn(0x7fda2c0b2fc0 msgr2=0x7fda2c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.354+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 shutdown_connections 2023-11-08T08:20:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.355+0000 7fda33378700 1 -- 172.21.15.116:0/779948287 wait complete. 2023-11-08T08:20:33.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.486+0000 7f6f4e0de700 1 Processor -- start 2023-11-08T08:20:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.486+0000 7f6f4e0de700 1 -- start start 2023-11-08T08:20:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f4e0de700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4806aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f4e0de700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f480cd5a0 0x7f6f48160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f4e0de700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f480d1ee0 con 0x7f6f480b22a0 2023-11-08T08:20:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f4e0de700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4806b420 con 0x7f6f480cd5a0 2023-11-08T08:20:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f4e0de700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480cdb10 con 0x7f6f4806ab10 2023-11-08T08:20:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f3ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f480cd5a0 0x7f6f48160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f3ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f480cd5a0 0x7f6f48160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41852/0 (socket says 172.21.15.116:41852) 2023-11-08T08:20:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f3ffff700 1 -- 172.21.15.116:0/2862927476 learned_addr learned my addr 172.21.15.116:0/2862927476 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.487+0000 7f6f477fe700 1 --2- 172.21.15.116:0/2862927476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4806aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f3ffff700 1 -- 172.21.15.116:0/2862927476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 msgr2=0x7f6f4806aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f3ffff700 1 --2- 172.21.15.116:0/2862927476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4806aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f3ffff700 1 -- 172.21.15.116:0/2862927476 >> v1:172.21.15.129:6789/0 conn(0x7f6f480b22a0 legacy=0x7f6f48160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f3ffff700 1 -- 172.21.15.116:0/2862927476 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f480d1890 con 0x7f6f480cd5a0 2023-11-08T08:20:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f3ffff700 1 --2- 172.21.15.116:0/2862927476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f48160300 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6f300098f0 tx=0x7f6f30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff84ca48cfe9ac56 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f46ffd700 1 -- 172.21.15.116:0/2862927476 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30011830 con 0x7f6f480cd5a0 2023-11-08T08:20:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f46ffd700 1 -- 172.21.15.116:0/2862927476 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f30011990 con 0x7f6f480cd5a0 2023-11-08T08:20:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f46ffd700 1 -- 172.21.15.116:0/2862927476 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30011d20 con 0x7f6f480cd5a0 2023-11-08T08:20:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2862927476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 msgr2=0x7f6f48160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2862927476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f48160300 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6f300098f0 tx=0x7f6f30009c00 comp rx=0 tx=0).stop 2023-11-08T08:20:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2862927476 shutdown_connections 2023-11-08T08:20:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.488+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2862927476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4806aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.489+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2862927476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f48160300 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.489+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2862927476 >> 172.21.15.116:0/2862927476 conn(0x7f6f4806c520 msgr2=0x7f6f4806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.489+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2862927476 shutdown_connections 2023-11-08T08:20:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.489+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2862927476 wait complete. 2023-11-08T08:20:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.490+0000 7f6f4e0de700 1 Processor -- start 2023-11-08T08:20:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.490+0000 7f6f4e0de700 1 -- start start 2023-11-08T08:20:33.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.490+0000 7f6f4e0de700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.490+0000 7f6f4e0de700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f480b22a0 0x7f6f480c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.490+0000 7f6f4e0de700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f480c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.490+0000 7f6f4e0de700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4814f730 con 0x7f6f480cd5a0 2023-11-08T08:20:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f4e0de700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481508e0 con 0x7f6f480b22a0 2023-11-08T08:20:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f4e0de700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f48150bc0 con 0x7f6f4806ab10 2023-11-08T08:20:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f480c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f480c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41862/0 (socket says 172.21.15.116:41862) 2023-11-08T08:20:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 -- 172.21.15.116:0/2510076494 learned_addr learned my addr 172.21.15.116:0/2510076494 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f477fe700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f3ffff700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f480b22a0 0x7f6f480c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 -- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 msgr2=0x7f6f4814f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4814f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 -- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f480b22a0 msgr2=0x7f6f480c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f480b22a0 0x7f6f480c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 -- 172.21.15.116:0/2510076494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30009550 con 0x7f6f480cd5a0 2023-11-08T08:20:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f47fff700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f480c8b10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000cc50 tx=0x7f6f4000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40003c00 con 0x7f6f480cd5a0 2023-11-08T08:20:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f40003d60 con 0x7f6f480cd5a0 2023-11-08T08:20:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4815cad0 con 0x7f6f480cd5a0 2023-11-08T08:20:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000aab0 con 0x7f6f480cd5a0 2023-11-08T08:20:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.491+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f480d1b70 con 0x7f6f480cd5a0 2023-11-08T08:20:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.492+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f480b1ec0 con 0x7f6f480cd5a0 2023-11-08T08:20:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.493+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f40010050 con 0x7f6f480cd5a0 2023-11-08T08:20:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.493+0000 7f6f44ff9700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6f280682f0 0x7f6f2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.493+0000 7f6f477fe700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6f280682f0 0x7f6f2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.493+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6f40085ac0 con 0x7f6f480cd5a0 2023-11-08T08:20:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.493+0000 7f6f477fe700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6f280682f0 0x7f6f2806a7a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009b40 tx=0x7f6f38009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.497+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f40052320 con 0x7f6f480cd5a0 2023-11-08T08:20:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.681+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f480743f0 con 0x7f6f480cd5a0 2023-11-08T08:20:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.682+0000 7f6f44ff9700 1 -- 172.21.15.116:0/2510076494 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17922 (secure 0 0 0) 0x7f6f40055940 con 0x7f6f480cd5a0 2023-11-08T08:20:33.684 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:33.684 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":43,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:33.012466+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6f280682f0 msgr2=0x7f6f2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6f280682f0 0x7f6f2806a7a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009b40 tx=0x7f6f38009210 comp rx=0 tx=0).stop 2023-11-08T08:20:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 msgr2=0x7f6f480c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f480c8b10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000cc50 tx=0x7f6f4000cf60 comp rx=0 tx=0).stop 2023-11-08T08:20:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 shutdown_connections 2023-11-08T08:20:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f480cd5a0 0x7f6f480c8b10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6f4806ab10 0x7f6f4814f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6f280682f0 0x7f6f2806a7a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 --2- 172.21.15.116:0/2510076494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f480b22a0 0x7f6f480c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 >> 172.21.15.116:0/2510076494 conn(0x7f6f4806c520 msgr2=0x7f6f480b49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:33.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.684+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 shutdown_connections 2023-11-08T08:20:33.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.685+0000 7f6f4e0de700 1 -- 172.21.15.116:0/2510076494 wait complete. 2023-11-08T08:20:33.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-08T08:20:33.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.820+0000 7fe2a08f0700 1 Processor -- start 2023-11-08T08:20:33.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.820+0000 7fe2a08f0700 1 -- start start 2023-11-08T08:20:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe2a08f0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe29c159710 0x7fe29c070370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe2a08f0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c0cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe2a08f0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe29c0c9a50 0x7fe29c0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe2a08f0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29c0cf290 con 0x7fe29c0c9a50 2023-11-08T08:20:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe2a08f0700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29c0ca410 con 0x7fe29c159710 2023-11-08T08:20:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe2a08f0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c06a310 con 0x7fe29c0cee40 2023-11-08T08:20:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe29c0c9a50 0x7fe29c0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe29c0c9a50 0x7fe29c0c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41872/0 (socket says 172.21.15.116:41872) 2023-11-08T08:20:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 -- 172.21.15.116:0/3946483553 learned_addr learned my addr 172.21.15.116:0/3946483553 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe299d9b700 1 --2- 172.21.15.116:0/3946483553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c0cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29a59c700 1 --2- 172.21.15.116:0/3946483553 >> v2:172.21.15.129:3300/0 conn(0x7fe29c159710 0x7fe29c070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 -- 172.21.15.116:0/3946483553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 msgr2=0x7fe29c0cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 --2- 172.21.15.116:0/3946483553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c0cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 -- 172.21.15.116:0/3946483553 >> v2:172.21.15.129:3300/0 conn(0x7fe29c159710 msgr2=0x7fe29c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 --2- 172.21.15.116:0/3946483553 >> v2:172.21.15.129:3300/0 conn(0x7fe29c159710 0x7fe29c070370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 -- 172.21.15.116:0/3946483553 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0ce530 con 0x7fe29c0c9a50 2023-11-08T08:20:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.821+0000 7fe29ad9d700 1 --2- 172.21.15.116:0/3946483553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c0c9ed0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe29c06fc50 tx=0x7fe29400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908eac35cb5fc96f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe29959a700 1 -- 172.21.15.116:0/3946483553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29400e070 con 0x7fe29c0c9a50 2023-11-08T08:20:33.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe29959a700 1 -- 172.21.15.116:0/3946483553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29400b910 con 0x7fe29c0c9a50 2023-11-08T08:20:33.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe29959a700 1 -- 172.21.15.116:0/3946483553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29400bcc0 con 0x7fe29c0c9a50 2023-11-08T08:20:33.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 -- 172.21.15.116:0/3946483553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 msgr2=0x7fe29c0c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/3946483553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c0c9ed0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe29c06fc50 tx=0x7fe29400b330 comp rx=0 tx=0).stop 2023-11-08T08:20:33.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 -- 172.21.15.116:0/3946483553 shutdown_connections 2023-11-08T08:20:33.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/3946483553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c0c9ed0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/3946483553 >> v2:172.21.15.129:3300/0 conn(0x7fe29c159710 0x7fe29c070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/3946483553 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c0cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 -- 172.21.15.116:0/3946483553 >> 172.21.15.116:0/3946483553 conn(0x7fe29c0cf400 msgr2=0x7fe29c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:33.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 -- 172.21.15.116:0/3946483553 shutdown_connections 2023-11-08T08:20:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.822+0000 7fe2a08f0700 1 -- 172.21.15.116:0/3946483553 wait complete. 2023-11-08T08:20:33.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.823+0000 7fe2a08f0700 1 Processor -- start 2023-11-08T08:20:33.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 -- start start 2023-11-08T08:20:33.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29c159710 0x7fe29c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c0c9140 con 0x7fe29c0c9a50 2023-11-08T08:20:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c148fb0 con 0x7fe29c159710 2023-11-08T08:20:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe2a08f0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c1464e0 con 0x7fe29c0cee40 2023-11-08T08:20:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe29a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe29a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c148a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41884/0 (socket says 172.21.15.116:41884) 2023-11-08T08:20:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.824+0000 7fe29a59c700 1 -- 172.21.15.116:0/719725973 learned_addr learned my addr 172.21.15.116:0/719725973 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe299d9b700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29ad9d700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29c159710 0x7fe29c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29a59c700 1 -- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 msgr2=0x7fe29c15bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29a59c700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c15bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29a59c700 1 -- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29c159710 msgr2=0x7fe29c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29a59c700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29c159710 0x7fe29c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29a59c700 1 -- 172.21.15.116:0/719725973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe294009b30 con 0x7fe29c0c9a50 2023-11-08T08:20:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe29a59c700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c148a70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00e810 tx=0x7fe28c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c013610 con 0x7fe29c0c9a50 2023-11-08T08:20:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28c013770 con 0x7fe29c0c9a50 2023-11-08T08:20:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c013b20 con 0x7fe29c0c9a50 2023-11-08T08:20:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c06a5b0 con 0x7fe29c0c9a50 2023-11-08T08:20:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.825+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c0ce7e0 con 0x7fe29c0c9a50 2023-11-08T08:20:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.826+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c06a890 con 0x7fe29c0c9a50 2023-11-08T08:20:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.827+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe28c023070 con 0x7fe29c0c9a50 2023-11-08T08:20:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.827+0000 7fe28affd700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe27c0682f0 0x7fe27c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.827+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe28c085e30 con 0x7fe29c0c9a50 2023-11-08T08:20:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.828+0000 7fe299d9b700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe27c0682f0 0x7fe27c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.828+0000 7fe299d9b700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe27c0682f0 0x7fe27c06a7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe29c1f32f0 tx=0x7fe290008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:33.831+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe28c0525a0 con 0x7fe29c0c9a50 2023-11-08T08:20:34.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:34.055+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe29c0743f0 con 0x7fe29c0c9a50 2023-11-08T08:20:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.021+0000 7fe28affd700 1 -- 172.21.15.116:0/719725973 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe28c0210b0 con 0x7fe29c0c9a50 2023-11-08T08:20:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.023+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe27c0682f0 msgr2=0x7fe27c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.023+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe27c0682f0 0x7fe27c06a7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe29c1f32f0 tx=0x7fe290008040 comp rx=0 tx=0).stop 2023-11-08T08:20:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.023+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 msgr2=0x7fe29c148a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.023+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c148a70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00e810 tx=0x7fe28c00ebd0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 shutdown_connections 2023-11-08T08:20:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe27c0682f0 0x7fe27c06a7a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29c159710 0x7fe29c0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe29c0c9a50 0x7fe29c148a70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 --2- 172.21.15.116:0/719725973 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe29c0cee40 0x7fe29c15bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 >> 172.21.15.116:0/719725973 conn(0x7fe29c0cf400 msgr2=0x7fe29c0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 shutdown_connections 2023-11-08T08:20:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.024+0000 7fe2a08f0700 1 -- 172.21.15.116:0/719725973 wait complete. 2023-11-08T08:20:35.039 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-08T08:20:35.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-08T08:20:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.167+0000 7fecdc857700 1 Processor -- start 2023-11-08T08:20:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.167+0000 7fecdc857700 1 -- start start 2023-11-08T08:20:35.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecdc857700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 0x7fecd8160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecdc857700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fecd80ce740 0x7fecd81608c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecdc857700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd815cc70 con 0x7fecd815c8a0 2023-11-08T08:20:35.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecdc857700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd806c880 con 0x7fecd80ce740 2023-11-08T08:20:35.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecdc857700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd806cad0 con 0x7fecd806a980 2023-11-08T08:20:35.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecd659c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fecd815c8a0 0x7fecd806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52074/0 (socket says 172.21.15.116:52074) 2023-11-08T08:20:35.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecd659c700 1 -- 172.21.15.116:0/984390109 learned_addr learned my addr 172.21.15.116:0/984390109 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:35.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecd6d9d700 1 --2- 172.21.15.116:0/984390109 >> v2:172.21.15.129:3300/0 conn(0x7fecd80ce740 0x7fecd81608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981102121 0 0) 0x7fecd815cc70 con 0x7fecd815c8a0 2023-11-08T08:20:35.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 0x7fecd8160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.168+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc00035b0 con 0x7fecd815c8a0 2023-11-08T08:20:35.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991567014 0 0) 0x7fecc00035b0 con 0x7fecd815c8a0 2023-11-08T08:20:35.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 msgr2=0x7fecd8160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:35.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 0x7fecd8160380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 >> v2:172.21.15.129:3300/0 conn(0x7fecd80ce740 msgr2=0x7fecd81608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:35.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 --2- 172.21.15.116:0/984390109 >> v2:172.21.15.129:3300/0 conn(0x7fecd80ce740 0x7fecd81608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd806cdb0 con 0x7fecd815c8a0 2023-11-08T08:20:35.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7feccc001670 con 0x7fecd815c8a0 2023-11-08T08:20:35.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 >> v1:172.21.15.116:6789/0 conn(0x7fecd815c8a0 legacy=0x7fecd806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:35.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 0x7fecc0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecc0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecc0004d90 0x7fecc0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd806c880 con 0x7fecc0003f80 2023-11-08T08:20:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd806cad0 con 0x7fecc0004d90 2023-11-08T08:20:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000ba40 con 0x7fecd806a980 2023-11-08T08:20:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecc0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd659c700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 0x7fecc0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd6d9d700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecc0004d90 0x7fecc0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 -- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 msgr2=0x7fecc0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd806a980 0x7fecc0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 -- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecc0004d90 msgr2=0x7fecc0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecc0004d90 0x7fecc0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 -- 172.21.15.116:0/984390109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc00035b0 con 0x7fecc0003f80 2023-11-08T08:20:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecc0004850 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009f90 tx=0x7fecc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f49ac309e69ef72 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc800ebd0 con 0x7fecc0003f80 2023-11-08T08:20:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc800f040 con 0x7fecc0003f80 2023-11-08T08:20:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.169+0000 7fecd559a700 1 -- 172.21.15.116:0/984390109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8014480 con 0x7fecc0003f80 2023-11-08T08:20:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecdc857700 1 -- 172.21.15.116:0/984390109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 msgr2=0x7fecc0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecdc857700 1 --2- 172.21.15.116:0/984390109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecc0004850 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009f90 tx=0x7fecc0005880 comp rx=0 tx=0).stop 2023-11-08T08:20:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecd6d9d700 1 -- 172.21.15.116:0/984390109 reap_dead start 2023-11-08T08:20:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecdc857700 1 -- 172.21.15.116:0/984390109 shutdown_connections 2023-11-08T08:20:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecdc857700 1 -- 172.21.15.116:0/984390109 >> 172.21.15.116:0/984390109 conn(0x7fecd806c010 msgr2=0x7fecd80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecdc857700 1 -- 172.21.15.116:0/984390109 shutdown_connections 2023-11-08T08:20:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.170+0000 7fecdc857700 1 -- 172.21.15.116:0/984390109 wait complete. 2023-11-08T08:20:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.171+0000 7fecdc857700 1 Processor -- start 2023-11-08T08:20:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.171+0000 7fecdc857700 1 -- start start 2023-11-08T08:20:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecdc857700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecd8155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecdc857700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecd815ef50 0x7fecd815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecdc857700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd8145e30 0x7fecd815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecdc857700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd8154450 con 0x7fecc0003f80 2023-11-08T08:20:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecdc857700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd815eb00 con 0x7fecd815ef50 2023-11-08T08:20:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecdc857700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd81546d0 con 0x7fecd8145e30 2023-11-08T08:20:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecd8155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecd8155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41900/0 (socket says 172.21.15.116:41900) 2023-11-08T08:20:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 -- 172.21.15.116:0/4172382374 learned_addr learned my addr 172.21.15.116:0/4172382374 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecd815ef50 0x7fecd815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 -- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd8145e30 msgr2=0x7fecd815af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:35.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd8145e30 0x7fecd815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 -- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecd815ef50 msgr2=0x7fecd815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:35.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecd815ef50 0x7fecd815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:35.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.172+0000 7fecd659c700 1 -- 172.21.15.116:0/4172382374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd806d370 con 0x7fecc0003f80 2023-11-08T08:20:35.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.173+0000 7fecd659c700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecd8155490 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7feccc000c00 tx=0x7feccc0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:35.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.173+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc0077c0 con 0x7fecc0003f80 2023-11-08T08:20:35.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.173+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc00f850 con 0x7fecc0003f80 2023-11-08T08:20:35.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.173+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd806d090 con 0x7fecc0003f80 2023-11-08T08:20:35.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.173+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc00fc00 con 0x7fecc0003f80 2023-11-08T08:20:35.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.173+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd80cd430 con 0x7fecc0003f80 2023-11-08T08:20:35.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.174+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd80743f0 con 0x7fecc0003f80 2023-11-08T08:20:35.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.174+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feccc00fd60 con 0x7fecc0003f80 2023-11-08T08:20:35.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.174+0000 7fecc6ffd700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fecb8068320 0x7fecb806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:35.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.175+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feccc085d80 con 0x7fecc0003f80 2023-11-08T08:20:35.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.175+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fecb8068320 0x7fecb806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:35.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.175+0000 7fecd5d9b700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fecb8068320 0x7fecb806a7d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fecc800afd0 tx=0x7fecc8013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:35.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.179+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feccc0525e0 con 0x7fecc0003f80 2023-11-08T08:20:35.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:35.405+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fecd80b5b50 con 0x7fecc0003f80 2023-11-08T08:20:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.025+0000 7fecc6ffd700 1 -- 172.21.15.116:0/4172382374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7feccc087070 con 0x7fecc0003f80 2023-11-08T08:20:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fecb8068320 msgr2=0x7fecb806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fecb8068320 0x7fecb806a7d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fecc800afd0 tx=0x7fecc8013000 comp rx=0 tx=0).stop 2023-11-08T08:20:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 msgr2=0x7fecd8155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecd8155490 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7feccc000c00 tx=0x7feccc0076a0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 shutdown_connections 2023-11-08T08:20:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fecd8145e30 0x7fecd815af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0003f80 0x7fecd8155490 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fecb8068320 0x7fecb806a7d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 --2- 172.21.15.116:0/4172382374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fecd815ef50 0x7fecd815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 >> 172.21.15.116:0/4172382374 conn(0x7fecd806c010 msgr2=0x7fecd8159e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:36.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 shutdown_connections 2023-11-08T08:20:36.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.028+0000 7fecdc857700 1 -- 172.21.15.116:0/4172382374 wait complete. 2023-11-08T08:20:36.043 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-08T08:20:36.044 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.167+0000 7fcbb446e700 1 Processor -- start 2023-11-08T08:20:36.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.167+0000 7fcbb446e700 1 -- start start 2023-11-08T08:20:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.168+0000 7fcbb446e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcbac0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.168+0000 7fcbb446e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac0d1ee0 con 0x7fcbac0b22a0 2023-11-08T08:20:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.168+0000 7fcbb446e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac0cdb10 con 0x7fcbac06ab10 2023-11-08T08:20:36.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.168+0000 7fcbb446e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0d1890 con 0x7fcbac0cd5a0 2023-11-08T08:20:36.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.168+0000 7fcbb2a0b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcbac0b22a0 0x7fcbac0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52090/0 (socket says 172.21.15.116:52090) 2023-11-08T08:20:36.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.168+0000 7fcbb2a0b700 1 -- 172.21.15.116:0/2149824200 learned_addr learned my addr 172.21.15.116:0/2149824200 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394052397 0 0) 0x7fcbac0d1ee0 con 0x7fcbac0b22a0 2023-11-08T08:20:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb980035b0 con 0x7fcbac0b22a0 2023-11-08T08:20:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcbac0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706333272 0 0) 0x7fcb980035b0 con 0x7fcbac0b22a0 2023-11-08T08:20:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 msgr2=0x7fcbac0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcbac0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 >> v1:172.21.15.129:6789/0 conn(0x7fcbac06ab10 legacy=0x7fcbac06aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0d1b70 con 0x7fcbac0b22a0 2023-11-08T08:20:36.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fcba40019f0 con 0x7fcbac0b22a0 2023-11-08T08:20:36.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 >> v1:172.21.15.116:6789/0 conn(0x7fcbac0b22a0 legacy=0x7fcbac0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcb98003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb98004060 0x7fcb98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004d90 0x7fcb98005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0d1890 con 0x7fcb98004060 2023-11-08T08:20:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9800bba0 con 0x7fcb98004d90 2023-11-08T08:20:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9800bd60 con 0x7fcbac0cd5a0 2023-11-08T08:20:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb98004060 0x7fcb98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb220a700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcb98003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb2a0b700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004d90 0x7fcb98005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 -- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 msgr2=0x7fcb98003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcb98003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 -- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004d90 msgr2=0x7fcb98005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004d90 0x7fcb98005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 -- 172.21.15.116:0/2149824200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980035b0 con 0x7fcb98004060 2023-11-08T08:20:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb220a700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac0cd5a0 0x7fcb98003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb98004060 0x7fcb98004850 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009890 tx=0x7fcb98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd02cca8518cd566 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.169+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba800c070 con 0x7fcb98004060 2023-11-08T08:20:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba8013610 con 0x7fcb98004060 2023-11-08T08:20:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb1208700 1 -- 172.21.15.116:0/2149824200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba80139a0 con 0x7fcb98004060 2023-11-08T08:20:36.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb446e700 1 -- 172.21.15.116:0/2149824200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb98004060 msgr2=0x7fcb98004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb446e700 1 --2- 172.21.15.116:0/2149824200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb98004060 0x7fcb98004850 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009890 tx=0x7fcb98005880 comp rx=0 tx=0).stop 2023-11-08T08:20:36.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb2a0b700 1 -- 172.21.15.116:0/2149824200 reap_dead start 2023-11-08T08:20:36.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb446e700 1 -- 172.21.15.116:0/2149824200 shutdown_connections 2023-11-08T08:20:36.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb446e700 1 -- 172.21.15.116:0/2149824200 >> 172.21.15.116:0/2149824200 conn(0x7fcbac06c520 msgr2=0x7fcbac06c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:36.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb446e700 1 -- 172.21.15.116:0/2149824200 shutdown_connections 2023-11-08T08:20:36.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.170+0000 7fcbb446e700 1 -- 172.21.15.116:0/2149824200 wait complete. 2023-11-08T08:20:36.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.171+0000 7fcbb446e700 1 Processor -- start 2023-11-08T08:20:36.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.171+0000 7fcbb446e700 1 -- start start 2023-11-08T08:20:36.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb446e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004060 0x7fcbac0ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb446e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 0x7fcbac15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb446e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac15f370 0x7fcbac0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb446e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0cb3a0 con 0x7fcbac0c9210 2023-11-08T08:20:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb446e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac15f7f0 con 0x7fcb98004060 2023-11-08T08:20:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb446e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0ca720 con 0x7fcbac15f370 2023-11-08T08:20:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 0x7fcbac15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 0x7fcbac15ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41918/0 (socket says 172.21.15.116:41918) 2023-11-08T08:20:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 -- 172.21.15.116:0/1557365981 learned_addr learned my addr 172.21.15.116:0/1557365981 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb2a0b700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac15f370 0x7fcbac0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 -- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac15f370 msgr2=0x7fcbac0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac15f370 0x7fcbac0cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 -- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004060 msgr2=0x7fcbac0ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004060 0x7fcbac0ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 -- 172.21.15.116:0/1557365981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0b1ec0 con 0x7fcbac0c9210 2023-11-08T08:20:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.172+0000 7fcbb1a09700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 0x7fcbac15ee30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcba8016040 tx=0x7fcba80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.173+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba801c6f0 con 0x7fcbac0c9210 2023-11-08T08:20:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.173+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba801c870 con 0x7fcbac0c9210 2023-11-08T08:20:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.173+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac15cad0 con 0x7fcbac0c9210 2023-11-08T08:20:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.173+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba801cc20 con 0x7fcbac0c9210 2023-11-08T08:20:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.173+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac0b2050 con 0x7fcbac0c9210 2023-11-08T08:20:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.173+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac0743f0 con 0x7fcbac0c9210 2023-11-08T08:20:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.174+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcba802c030 con 0x7fcbac0c9210 2023-11-08T08:20:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.174+0000 7fcba2ffd700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcb900682f0 0x7fcb9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.174+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcba8086920 con 0x7fcbac0c9210 2023-11-08T08:20:36.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.174+0000 7fcbb220a700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcb900682f0 0x7fcb9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.175+0000 7fcbb220a700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcb900682f0 0x7fcb9006a7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007b30 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.178+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcba8053180 con 0x7fcbac0c9210 2023-11-08T08:20:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.401+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbac0d2c80 con 0x7fcbac0c9210 2023-11-08T08:20:36.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.402+0000 7fcba2ffd700 1 -- 172.21.15.116:0/1557365981 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fcba80567a0 con 0x7fcbac0c9210 2023-11-08T08:20:36.403 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:36.403 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:20:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcb900682f0 msgr2=0x7fcb9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcb900682f0 0x7fcb9006a7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007b30 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2023-11-08T08:20:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 msgr2=0x7fcbac15ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 0x7fcbac15ee30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcba8016040 tx=0x7fcba80047c0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 shutdown_connections 2023-11-08T08:20:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcbac15f370 0x7fcbac0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcb900682f0 0x7fcb9006a7a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb98004060 0x7fcbac0ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 --2- 172.21.15.116:0/1557365981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbac0c9210 0x7fcbac15ee30 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 >> 172.21.15.116:0/1557365981 conn(0x7fcbac06c520 msgr2=0x7fcbac0b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 shutdown_connections 2023-11-08T08:20:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.404+0000 7fcbb446e700 1 -- 172.21.15.116:0/1557365981 wait complete. 2023-11-08T08:20:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:20:36.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:20:36.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.536+0000 7f54003c2700 1 Processor -- start 2023-11-08T08:20:36.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.536+0000 7f54003c2700 1 -- start start 2023-11-08T08:20:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f54003c2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f806a270 0x7f53f806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f54003c2700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f53f806e570 0x7f53f8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f54003c2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f806ab80 con 0x7f53f815cc80 2023-11-08T08:20:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f54003c2700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f80b2490 con 0x7f53f806e570 2023-11-08T08:20:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f54003c2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f806e9b0 con 0x7f53f806a270 2023-11-08T08:20:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe15e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f806a270 0x7f53f806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fd95d700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f53f815cc80 0x7f53f806de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:39758/0 (socket says 172.21.15.116:39758) 2023-11-08T08:20:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fd95d700 1 -- 172.21.15.116:0/3160393646 learned_addr learned my addr 172.21.15.116:0/3160393646 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe95f700 1 --2- 172.21.15.116:0/3160393646 >> v2:172.21.15.129:3300/0 conn(0x7f53f806e570 0x7f53f8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fd15c700 1 -- 172.21.15.116:0/3160393646 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005542757 0 0) 0x7f53f806ab80 con 0x7f53f815cc80 2023-11-08T08:20:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fd15c700 1 -- 172.21.15.116:0/3160393646 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e40035b0 con 0x7f53f815cc80 2023-11-08T08:20:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe15e700 1 -- 172.21.15.116:0/3160393646 >> v2:172.21.15.129:3300/0 conn(0x7f53f806e570 msgr2=0x7f53f8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe15e700 1 --2- 172.21.15.116:0/3160393646 >> v2:172.21.15.129:3300/0 conn(0x7f53f806e570 0x7f53f8070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe15e700 1 -- 172.21.15.116:0/3160393646 >> v1:172.21.15.129:6789/0 conn(0x7f53f815cc80 legacy=0x7f53f806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe15e700 1 -- 172.21.15.116:0/3160393646 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f80b1810 con 0x7f53f806a270 2023-11-08T08:20:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.537+0000 7f53fe95f700 1 --2- 172.21.15.116:0/3160393646 >> v2:172.21.15.129:3300/0 conn(0x7f53f806e570 0x7f53f8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f53fe15e700 1 --2- 172.21.15.116:0/3160393646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f806a270 0x7f53f806a640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009840 tx=0x7f53f4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6212afa5e98080b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f53fd15c700 1 -- 172.21.15.116:0/3160393646 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4011830 con 0x7f53f806a270 2023-11-08T08:20:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f53fd15c700 1 -- 172.21.15.116:0/3160393646 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f4011990 con 0x7f53f806a270 2023-11-08T08:20:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f53fd15c700 1 -- 172.21.15.116:0/3160393646 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4011d00 con 0x7f53f806a270 2023-11-08T08:20:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 -- 172.21.15.116:0/3160393646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f806a270 msgr2=0x7f53f806a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 --2- 172.21.15.116:0/3160393646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f806a270 0x7f53f806a640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009840 tx=0x7f53f4009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 -- 172.21.15.116:0/3160393646 shutdown_connections 2023-11-08T08:20:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 --2- 172.21.15.116:0/3160393646 >> v2:172.21.15.129:3300/0 conn(0x7f53f806e570 0x7f53f8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 --2- 172.21.15.116:0/3160393646 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f806a270 0x7f53f806a640 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 -- 172.21.15.116:0/3160393646 >> 172.21.15.116:0/3160393646 conn(0x7f53f806c010 msgr2=0x7f53f80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 -- 172.21.15.116:0/3160393646 shutdown_connections 2023-11-08T08:20:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.538+0000 7f54003c2700 1 -- 172.21.15.116:0/3160393646 wait complete. 2023-11-08T08:20:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 Processor -- start 2023-11-08T08:20:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 -- start start 2023-11-08T08:20:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 0x7f53f80cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53f806e570 0x7f53f8152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f815cc80 0x7f53f80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f80cbd10 con 0x7f53f806a270 2023-11-08T08:20:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f80c7110 con 0x7f53f806e570 2023-11-08T08:20:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f54003c2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f80cad30 con 0x7f53f815cc80 2023-11-08T08:20:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.540+0000 7f53fe15e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 0x7f53f80cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 0x7f53f80cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41932/0 (socket says 172.21.15.116:41932) 2023-11-08T08:20:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 -- 172.21.15.116:0/1798103615 learned_addr learned my addr 172.21.15.116:0/1798103615 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 -- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f815cc80 msgr2=0x7f53f80b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f815cc80 0x7f53f80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 -- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53f806e570 msgr2=0x7f53f8152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fd95d700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53f806e570 0x7f53f8152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53f806e570 0x7f53f8152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 -- 172.21.15.116:0/1798103615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0009a70 con 0x7f53f806a270 2023-11-08T08:20:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53fe15e700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 0x7f53f80cb7d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009fd0 tx=0x7f53f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4004550 con 0x7f53f806a270 2023-11-08T08:20:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f401e070 con 0x7f53f806a270 2023-11-08T08:20:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f40094a0 con 0x7f53f806a270 2023-11-08T08:20:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40038c0 con 0x7f53f806a270 2023-11-08T08:20:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.541+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f80b1dd0 con 0x7f53f806a270 2023-11-08T08:20:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.542+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f80b1af0 con 0x7f53f806a270 2023-11-08T08:20:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.542+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f53f400f050 con 0x7f53f806a270 2023-11-08T08:20:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.543+0000 7f53eeffd700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53dc0682f0 0x7f53dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.543+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f53f4086ba0 con 0x7f53f806a270 2023-11-08T08:20:36.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.543+0000 7f53fd95d700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53dc0682f0 0x7f53dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.543+0000 7f53fd95d700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53dc0682f0 0x7f53dc06a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f53f80c6b80 tx=0x7f53e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.547+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53f4053380 con 0x7f53f806a270 2023-11-08T08:20:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.728+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53f80b20b0 con 0x7f53f806a270 2023-11-08T08:20:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.729+0000 7f53eeffd700 1 -- 172.21.15.116:0/1798103615 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+17909 (secure 0 0 0) 0x7f53f4020090 con 0x7f53f806a270 2023-11-08T08:20:36.731 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:36.731 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":44,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:34.014710+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:20:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53dc0682f0 msgr2=0x7f53dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53dc0682f0 0x7f53dc06a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f53f80c6b80 tx=0x7f53e8009040 comp rx=0 tx=0).stop 2023-11-08T08:20:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 msgr2=0x7f53f80cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 0x7f53f80cb7d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009fd0 tx=0x7f53f4004750 comp rx=0 tx=0).stop 2023-11-08T08:20:36.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 shutdown_connections 2023-11-08T08:20:36.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f53f815cc80 0x7f53f80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f806a270 0x7f53f80cb7d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f53dc0682f0 0x7f53dc06a7a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 --2- 172.21.15.116:0/1798103615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53f806e570 0x7f53f8152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 >> 172.21.15.116:0/1798103615 conn(0x7f53f806c010 msgr2=0x7f53f80ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 shutdown_connections 2023-11-08T08:20:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.732+0000 7f54003c2700 1 -- 172.21.15.116:0/1798103615 wait complete. 2023-11-08T08:20:36.745 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-08T08:20:36.746 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:20:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.864+0000 7f62459e8700 1 Processor -- start 2023-11-08T08:20:36.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.864+0000 7f62459e8700 1 -- start start 2023-11-08T08:20:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f62459e8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 0x7f62400cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f62459e8700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400d1ee0 con 0x7f624006ab10 2023-11-08T08:20:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f62459e8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400cdb10 con 0x7f62400cd5a0 2023-11-08T08:20:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f62459e8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400d1890 con 0x7f62400b22a0 2023-11-08T08:20:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f623effd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f624006ab10 0x7f624006aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52100/0 (socket says 172.21.15.116:52100) 2023-11-08T08:20:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f623effd700 1 -- 172.21.15.116:0/1374495532 learned_addr learned my addr 172.21.15.116:0/1374495532 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868188073 0 0) 0x7f62400d1ee0 con 0x7f624006ab10 2023-11-08T08:20:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f623f7fe700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 0x7f62400cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.865+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62280035b0 con 0x7f624006ab10 2023-11-08T08:20:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1152036858 0 0) 0x7f62280035b0 con 0x7f624006ab10 2023-11-08T08:20:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 msgr2=0x7f62400cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 0x7f62400cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 >> v1:172.21.15.129:6789/0 conn(0x7f62400cd5a0 legacy=0x7f62400cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62400d1b70 con 0x7f624006ab10 2023-11-08T08:20:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f6234001670 con 0x7f624006ab10 2023-11-08T08:20:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 >> v1:172.21.15.116:6789/0 conn(0x7f624006ab10 legacy=0x7f624006aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228003a30 0x7f6228003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 0x7f6228004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6228004d90 0x7f6228005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400d1890 con 0x7f6228003a30 2023-11-08T08:20:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622800bbd0 con 0x7f6228004d90 2023-11-08T08:20:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622800bd90 con 0x7f62400b22a0 2023-11-08T08:20:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228003a30 0x7f6228003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623e7fc700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 0x7f6228004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623f7fe700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6228004d90 0x7f6228005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 -- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 msgr2=0x7f6228004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f62400b22a0 0x7f6228004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 -- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6228004d90 msgr2=0x7f6228005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6228004d90 0x7f6228005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 -- 172.21.15.116:0/1374495532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62280035b0 con 0x7f6228003a30 2023-11-08T08:20:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623effd700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228003a30 0x7f6228003e40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f623400c240 tx=0x7f6228005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3fbc279db756c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234007c10 con 0x7f6228003a30 2023-11-08T08:20:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623400cde0 con 0x7f6228003a30 2023-11-08T08:20:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.866+0000 7f623dffb700 1 -- 172.21.15.116:0/1374495532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234005ba0 con 0x7f6228003a30 2023-11-08T08:20:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f62459e8700 1 -- 172.21.15.116:0/1374495532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228003a30 msgr2=0x7f6228003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f62459e8700 1 --2- 172.21.15.116:0/1374495532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6228003a30 0x7f6228003e40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f623400c240 tx=0x7f6228005880 comp rx=0 tx=0).stop 2023-11-08T08:20:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f623f7fe700 1 -- 172.21.15.116:0/1374495532 reap_dead start 2023-11-08T08:20:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f62459e8700 1 -- 172.21.15.116:0/1374495532 shutdown_connections 2023-11-08T08:20:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f62459e8700 1 -- 172.21.15.116:0/1374495532 >> 172.21.15.116:0/1374495532 conn(0x7f624006c520 msgr2=0x7f624006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f62459e8700 1 -- 172.21.15.116:0/1374495532 shutdown_connections 2023-11-08T08:20:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.867+0000 7f62459e8700 1 -- 172.21.15.116:0/1374495532 wait complete. 2023-11-08T08:20:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.868+0000 7f62459e8700 1 Processor -- start 2023-11-08T08:20:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.869+0000 7f62459e8700 1 -- start start 2023-11-08T08:20:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6228003a30 0x7f624014ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 0x7f6240158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6240159110 0x7f6240159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400ca370 con 0x7f62400ccc70 2023-11-08T08:20:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400ca020 con 0x7f6240159110 2023-11-08T08:20:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6240150440 con 0x7f6228003a30 2023-11-08T08:20:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 0x7f6240158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 0x7f6240158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41950/0 (socket says 172.21.15.116:41950) 2023-11-08T08:20:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 -- 172.21.15.116:0/3295242781 learned_addr learned my addr 172.21.15.116:0/3295242781 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623effd700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6228003a30 0x7f624014ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623f7fe700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6240159110 0x7f6240159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 -- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6228003a30 msgr2=0x7f624014ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6228003a30 0x7f624014ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 -- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6240159110 msgr2=0x7f6240159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6240159110 0x7f6240159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 -- 172.21.15.116:0/3295242781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62400b1ec0 con 0x7f62400ccc70 2023-11-08T08:20:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623f7fe700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6240159110 0x7f6240159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f623e7fc700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 0x7f6240158bd0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f623000f850 tx=0x7f623000fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230005110 con 0x7f62400ccc70 2023-11-08T08:20:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6230005270 con 0x7f62400ccc70 2023-11-08T08:20:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624015cad0 con 0x7f62400ccc70 2023-11-08T08:20:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230006860 con 0x7f62400ccc70 2023-11-08T08:20:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.870+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62400b2050 con 0x7f62400ccc70 2023-11-08T08:20:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.871+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62400ca4b0 con 0x7f62400ccc70 2023-11-08T08:20:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.872+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f623001c030 con 0x7f62400ccc70 2023-11-08T08:20:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.872+0000 7f6227fff700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f622c0682f0 0x7f622c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.872+0000 7f623effd700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f622c0682f0 0x7f622c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.872+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f62300853e0 con 0x7f62400ccc70 2023-11-08T08:20:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.872+0000 7f623effd700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f622c0682f0 0x7f622c06a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6234015040 tx=0x7f6234007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:36.876+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f62400ca4b0 con 0x7f62400ccc70 2023-11-08T08:20:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.053+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62400ca160 con 0x7f622c0682f0 2023-11-08T08:20:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.062+0000 7f6227fff700 1 -- 172.21.15.116:0/3295242781 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311410 (secure 0 0 0) 0x7f6234007b30 con 0x7f622c0682f0 2023-11-08T08:20:37.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.065+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f622c0682f0 msgr2=0x7f622c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f622c0682f0 0x7f622c06a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6234015040 tx=0x7f6234007000 comp rx=0 tx=0).stop 2023-11-08T08:20:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 msgr2=0x7f6240158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 0x7f6240158bd0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f623000f850 tx=0x7f623000fc10 comp rx=0 tx=0).stop 2023-11-08T08:20:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 shutdown_connections 2023-11-08T08:20:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f622c0682f0 0x7f622c06a7a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6240159110 0x7f6240159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6228003a30 0x7f624014ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 --2- 172.21.15.116:0/3295242781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62400ccc70 0x7f6240158bd0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 >> 172.21.15.116:0/3295242781 conn(0x7f624006c520 msgr2=0x7f62400b5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:37.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 shutdown_connections 2023-11-08T08:20:37.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.066+0000 7f62459e8700 1 -- 172.21.15.116:0/3295242781 wait complete. 2023-11-08T08:20:37.072 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:20:37.079 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2023-11-08T08:20:36.377308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":73348,"kb_used_data":5252,"kb_used_omap":0,"kb_used_meta":68032,"kb_avail":754901372,"statfs":{"total":773094113280,"available":773019004928,"internally_reserved":0,"allocated":5378048,"data_stored":2943639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69664768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.755001"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089068+0000","last_change":"2023-11-08T08:20:33.040952+0000","last_active":"2023-11-08T08:20:34.089068+0000","last_peered":"2023-11-08T08:20:34.089068+0000","last_clean":"2023-11-08T08:20:34.089068+0000","last_became_active":"2023-11-08T08:20:33.040646+0000","last_became_peered":"2023-11-08T08:20:33.040646+0000","last_unstale":"2023-11-08T08:20:34.089068+0000","last_undegraded":"2023-11-08T08:20:34.089068+0000","last_fullsized":"2023-11-08T08:20:34.089068+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087995+0000","last_change":"2023-11-08T08:20:33.026201+0000","last_active":"2023-11-08T08:20:34.087995+0000","last_peered":"2023-11-08T08:20:34.087995+0000","last_clean":"2023-11-08T08:20:34.087995+0000","last_became_active":"2023-11-08T08:20:33.025933+0000","last_became_peered":"2023-11-08T08:20:33.025933+0000","last_unstale":"2023-11-08T08:20:34.087995+0000","last_undegraded":"2023-11-08T08:20:34.087995+0000","last_fullsized":"2023-11-08T08:20:34.087995+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088583+0000","last_change":"2023-11-08T08:20:33.039111+0000","last_active":"2023-11-08T08:20:34.088583+0000","last_peered":"2023-11-08T08:20:34.088583+0000","last_clean":"2023-11-08T08:20:34.088583+0000","last_became_active":"2023-11-08T08:20:33.038823+0000","last_became_peered":"2023-11-08T08:20:33.038823+0000","last_unstale":"2023-11-08T08:20:34.088583+0000","last_undegraded":"2023-11-08T08:20:34.088583+0000","last_fullsized":"2023-11-08T08:20:34.088583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:52:48.726062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089092+0000","last_change":"2023-11-08T08:20:33.033970+0000","last_active":"2023-11-08T08:20:34.089092+0000","last_peered":"2023-11-08T08:20:34.089092+0000","last_clean":"2023-11-08T08:20:34.089092+0000","last_became_active":"2023-11-08T08:20:33.033693+0000","last_became_peered":"2023-11-08T08:20:33.033693+0000","last_unstale":"2023-11-08T08:20:34.089092+0000","last_undegraded":"2023-11-08T08:20:34.089092+0000","last_fullsized":"2023-11-08T08:20:34.089092+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:46:36.745520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087772+0000","last_change":"2023-11-08T08:20:33.031233+0000","last_active":"2023-11-08T08:20:34.087772+0000","last_peered":"2023-11-08T08:20:34.087772+0000","last_clean":"2023-11-08T08:20:34.087772+0000","last_became_active":"2023-11-08T08:20:33.030959+0000","last_became_peered":"2023-11-08T08:20:33.030959+0000","last_unstale":"2023-11-08T08:20:34.087772+0000","last_undegraded":"2023-11-08T08:20:34.087772+0000","last_fullsized":"2023-11-08T08:20:34.087772+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:56:33.178296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021405+0000","last_change":"2023-11-08T08:20:33.039791+0000","last_active":"2023-11-08T08:20:34.021405+0000","last_peered":"2023-11-08T08:20:34.021405+0000","last_clean":"2023-11-08T08:20:34.021405+0000","last_became_active":"2023-11-08T08:20:33.039509+0000","last_became_peered":"2023-11-08T08:20:33.039509+0000","last_unstale":"2023-11-08T08:20:34.021405+0000","last_undegraded":"2023-11-08T08:20:34.021405+0000","last_fullsized":"2023-11-08T08:20:34.021405+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021786+0000","last_change":"2023-11-08T08:20:33.040580+0000","last_active":"2023-11-08T08:20:34.021786+0000","last_peered":"2023-11-08T08:20:34.021786+0000","last_clean":"2023-11-08T08:20:34.021786+0000","last_became_active":"2023-11-08T08:20:33.040334+0000","last_became_peered":"2023-11-08T08:20:33.040334+0000","last_unstale":"2023-11-08T08:20:34.021786+0000","last_undegraded":"2023-11-08T08:20:34.021786+0000","last_fullsized":"2023-11-08T08:20:34.021786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021173+0000","last_change":"2023-11-08T08:20:33.047781+0000","last_active":"2023-11-08T08:20:34.021173+0000","last_peered":"2023-11-08T08:20:34.021173+0000","last_clean":"2023-11-08T08:20:34.021173+0000","last_became_active":"2023-11-08T08:20:33.047519+0000","last_became_peered":"2023-11-08T08:20:33.047519+0000","last_unstale":"2023-11-08T08:20:34.021173+0000","last_undegraded":"2023-11-08T08:20:34.021173+0000","last_fullsized":"2023-11-08T08:20:34.021173+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088850+0000","last_change":"2023-11-08T08:20:33.038697+0000","last_active":"2023-11-08T08:20:34.088850+0000","last_peered":"2023-11-08T08:20:34.088850+0000","last_clean":"2023-11-08T08:20:34.088850+0000","last_became_active":"2023-11-08T08:20:33.038400+0000","last_became_peered":"2023-11-08T08:20:33.038400+0000","last_unstale":"2023-11-08T08:20:34.088850+0000","last_undegraded":"2023-11-08T08:20:34.088850+0000","last_fullsized":"2023-11-08T08:20:34.088850+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:01:14.330615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022078+0000","last_change":"2023-11-08T08:20:33.034382+0000","last_active":"2023-11-08T08:20:34.022078+0000","last_peered":"2023-11-08T08:20:34.022078+0000","last_clean":"2023-11-08T08:20:34.022078+0000","last_became_active":"2023-11-08T08:20:33.034087+0000","last_became_peered":"2023-11-08T08:20:33.034087+0000","last_unstale":"2023-11-08T08:20:34.022078+0000","last_undegraded":"2023-11-08T08:20:34.022078+0000","last_fullsized":"2023-11-08T08:20:34.022078+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088798+0000","last_change":"2023-11-08T08:20:33.039836+0000","last_active":"2023-11-08T08:20:34.088798+0000","last_peered":"2023-11-08T08:20:34.088798+0000","last_clean":"2023-11-08T08:20:34.088798+0000","last_became_active":"2023-11-08T08:20:33.039507+0000","last_became_peered":"2023-11-08T08:20:33.039507+0000","last_unstale":"2023-11-08T08:20:34.088798+0000","last_undegraded":"2023-11-08T08:20:34.088798+0000","last_fullsized":"2023-11-08T08:20:34.088798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021849+0000","last_change":"2023-11-08T08:20:33.039388+0000","last_active":"2023-11-08T08:20:34.021849+0000","last_peered":"2023-11-08T08:20:34.021849+0000","last_clean":"2023-11-08T08:20:34.021849+0000","last_became_active":"2023-11-08T08:20:33.039070+0000","last_became_peered":"2023-11-08T08:20:33.039070+0000","last_unstale":"2023-11-08T08:20:34.021849+0000","last_undegraded":"2023-11-08T08:20:34.021849+0000","last_fullsized":"2023-11-08T08:20:34.021849+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:08:02.296246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021517+0000","last_change":"2023-11-08T08:20:33.041545+0000","last_active":"2023-11-08T08:20:34.021517+0000","last_peered":"2023-11-08T08:20:34.021517+0000","last_clean":"2023-11-08T08:20:34.021517+0000","last_became_active":"2023-11-08T08:20:33.041264+0000","last_became_peered":"2023-11-08T08:20:33.041264+0000","last_unstale":"2023-11-08T08:20:34.021517+0000","last_undegraded":"2023-11-08T08:20:34.021517+0000","last_fullsized":"2023-11-08T08:20:34.021517+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088473+0000","last_change":"2023-11-08T08:20:33.046425+0000","last_active":"2023-11-08T08:20:34.088473+0000","last_peered":"2023-11-08T08:20:34.088473+0000","last_clean":"2023-11-08T08:20:34.088473+0000","last_became_active":"2023-11-08T08:20:33.046123+0000","last_became_peered":"2023-11-08T08:20:33.046123+0000","last_unstale":"2023-11-08T08:20:34.088473+0000","last_undegraded":"2023-11-08T08:20:34.088473+0000","last_fullsized":"2023-11-08T08:20:34.088473+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088187+0000","last_change":"2023-11-08T08:20:33.036896+0000","last_active":"2023-11-08T08:20:34.088187+0000","last_peered":"2023-11-08T08:20:34.088187+0000","last_clean":"2023-11-08T08:20:34.088187+0000","last_became_active":"2023-11-08T08:20:33.036462+0000","last_became_peered":"2023-11-08T08:20:33.036462+0000","last_unstale":"2023-11-08T08:20:34.088187+0000","last_undegraded":"2023-11-08T08:20:34.088187+0000","last_fullsized":"2023-11-08T08:20:34.088187+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:11:16.998268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021173+0000","last_change":"2023-11-08T08:20:33.040515+0000","last_active":"2023-11-08T08:20:34.021173+0000","last_peered":"2023-11-08T08:20:34.021173+0000","last_clean":"2023-11-08T08:20:34.021173+0000","last_became_active":"2023-11-08T08:20:33.040221+0000","last_became_peered":"2023-11-08T08:20:33.040221+0000","last_unstale":"2023-11-08T08:20:34.021173+0000","last_undegraded":"2023-11-08T08:20:34.021173+0000","last_fullsized":"2023-11-08T08:20:34.021173+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089214+0000","last_change":"2023-11-08T08:20:33.037114+0000","last_active":"2023-11-08T08:20:34.089214+0000","last_peered":"2023-11-08T08:20:34.089214+0000","last_clean":"2023-11-08T08:20:34.089214+0000","last_became_active":"2023-11-08T08:20:33.036834+0000","last_became_peered":"2023-11-08T08:20:33.036834+0000","last_unstale":"2023-11-08T08:20:34.089214+0000","last_undegraded":"2023-11-08T08:20:34.089214+0000","last_fullsized":"2023-11-08T08:20:34.089214+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088528+0000","last_change":"2023-11-08T08:20:33.037687+0000","last_active":"2023-11-08T08:20:34.088528+0000","last_peered":"2023-11-08T08:20:34.088528+0000","last_clean":"2023-11-08T08:20:34.088528+0000","last_became_active":"2023-11-08T08:20:33.035179+0000","last_became_peered":"2023-11-08T08:20:33.035179+0000","last_unstale":"2023-11-08T08:20:34.088528+0000","last_undegraded":"2023-11-08T08:20:34.088528+0000","last_fullsized":"2023-11-08T08:20:34.088528+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:07:01.555604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088706+0000","last_change":"2023-11-08T08:20:33.034002+0000","last_active":"2023-11-08T08:20:34.088706+0000","last_peered":"2023-11-08T08:20:34.088706+0000","last_clean":"2023-11-08T08:20:34.088706+0000","last_became_active":"2023-11-08T08:20:33.033684+0000","last_became_peered":"2023-11-08T08:20:33.033684+0000","last_unstale":"2023-11-08T08:20:34.088706+0000","last_undegraded":"2023-11-08T08:20:34.088706+0000","last_fullsized":"2023-11-08T08:20:34.088706+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:45:41.548671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088009+0000","last_change":"2023-11-08T08:20:33.027046+0000","last_active":"2023-11-08T08:20:34.088009+0000","last_peered":"2023-11-08T08:20:34.088009+0000","last_clean":"2023-11-08T08:20:34.088009+0000","last_became_active":"2023-11-08T08:20:33.026791+0000","last_became_peered":"2023-11-08T08:20:33.026791+0000","last_unstale":"2023-11-08T08:20:34.088009+0000","last_undegraded":"2023-11-08T08:20:34.088009+0000","last_fullsized":"2023-11-08T08:20:34.088009+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088410+0000","last_change":"2023-11-08T08:20:33.037485+0000","last_active":"2023-11-08T08:20:34.088410+0000","last_peered":"2023-11-08T08:20:34.088410+0000","last_clean":"2023-11-08T08:20:34.088410+0000","last_became_active":"2023-11-08T08:20:33.035255+0000","last_became_peered":"2023-11-08T08:20:33.035255+0000","last_unstale":"2023-11-08T08:20:34.088410+0000","last_undegraded":"2023-11-08T08:20:34.088410+0000","last_fullsized":"2023-11-08T08:20:34.088410+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:10:47.744346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088785+0000","last_change":"2023-11-08T08:20:33.044479+0000","last_active":"2023-11-08T08:20:34.088785+0000","last_peered":"2023-11-08T08:20:34.088785+0000","last_clean":"2023-11-08T08:20:34.088785+0000","last_became_active":"2023-11-08T08:20:33.044208+0000","last_became_peered":"2023-11-08T08:20:33.044208+0000","last_unstale":"2023-11-08T08:20:34.088785+0000","last_undegraded":"2023-11-08T08:20:34.088785+0000","last_fullsized":"2023-11-08T08:20:34.088785+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:32:46.187671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088263+0000","last_change":"2023-11-08T08:20:33.034864+0000","last_active":"2023-11-08T08:20:34.088263+0000","last_peered":"2023-11-08T08:20:34.088263+0000","last_clean":"2023-11-08T08:20:34.088263+0000","last_became_active":"2023-11-08T08:20:33.033759+0000","last_became_peered":"2023-11-08T08:20:33.033759+0000","last_unstale":"2023-11-08T08:20:34.088263+0000","last_undegraded":"2023-11-08T08:20:34.088263+0000","last_fullsized":"2023-11-08T08:20:34.088263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:01:06.860105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088211+0000","last_change":"2023-11-08T08:20:33.045243+0000","last_active":"2023-11-08T08:20:34.088211+0000","last_peered":"2023-11-08T08:20:34.088211+0000","last_clean":"2023-11-08T08:20:34.088211+0000","last_became_active":"2023-11-08T08:20:33.044970+0000","last_became_peered":"2023-11-08T08:20:33.044970+0000","last_unstale":"2023-11-08T08:20:34.088211+0000","last_undegraded":"2023-11-08T08:20:34.088211+0000","last_fullsized":"2023-11-08T08:20:34.088211+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:45:11.193724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089080+0000","last_change":"2023-11-08T08:20:33.046532+0000","last_active":"2023-11-08T08:20:34.089080+0000","last_peered":"2023-11-08T08:20:34.089080+0000","last_clean":"2023-11-08T08:20:34.089080+0000","last_became_active":"2023-11-08T08:20:33.046182+0000","last_became_peered":"2023-11-08T08:20:33.046182+0000","last_unstale":"2023-11-08T08:20:34.089080+0000","last_undegraded":"2023-11-08T08:20:34.089080+0000","last_fullsized":"2023-11-08T08:20:34.089080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088970+0000","last_change":"2023-11-08T08:20:33.044587+0000","last_active":"2023-11-08T08:20:34.088970+0000","last_peered":"2023-11-08T08:20:34.088970+0000","last_clean":"2023-11-08T08:20:34.088970+0000","last_became_active":"2023-11-08T08:20:33.044033+0000","last_became_peered":"2023-11-08T08:20:33.044033+0000","last_unstale":"2023-11-08T08:20:34.088970+0000","last_undegraded":"2023-11-08T08:20:34.088970+0000","last_fullsized":"2023-11-08T08:20:34.088970+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:43:40.912492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022278+0000","last_change":"2023-11-08T08:20:33.047351+0000","last_active":"2023-11-08T08:20:34.022278+0000","last_peered":"2023-11-08T08:20:34.022278+0000","last_clean":"2023-11-08T08:20:34.022278+0000","last_became_active":"2023-11-08T08:20:33.046969+0000","last_became_peered":"2023-11-08T08:20:33.046969+0000","last_unstale":"2023-11-08T08:20:34.022278+0000","last_undegraded":"2023-11-08T08:20:34.022278+0000","last_fullsized":"2023-11-08T08:20:34.022278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087934+0000","last_change":"2023-11-08T08:20:33.044167+0000","last_active":"2023-11-08T08:20:34.087934+0000","last_peered":"2023-11-08T08:20:34.087934+0000","last_clean":"2023-11-08T08:20:34.087934+0000","last_became_active":"2023-11-08T08:20:33.043978+0000","last_became_peered":"2023-11-08T08:20:33.043978+0000","last_unstale":"2023-11-08T08:20:34.087934+0000","last_undegraded":"2023-11-08T08:20:34.087934+0000","last_fullsized":"2023-11-08T08:20:34.087934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088438+0000","last_change":"2023-11-08T08:20:33.038706+0000","last_active":"2023-11-08T08:20:34.088438+0000","last_peered":"2023-11-08T08:20:34.088438+0000","last_clean":"2023-11-08T08:20:34.088438+0000","last_became_active":"2023-11-08T08:20:33.038438+0000","last_became_peered":"2023-11-08T08:20:33.038438+0000","last_unstale":"2023-11-08T08:20:34.088438+0000","last_undegraded":"2023-11-08T08:20:34.088438+0000","last_fullsized":"2023-11-08T08:20:34.088438+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021093+0000","last_change":"2023-11-08T08:20:33.037840+0000","last_active":"2023-11-08T08:20:34.021093+0000","last_peered":"2023-11-08T08:20:34.021093+0000","last_clean":"2023-11-08T08:20:34.021093+0000","last_became_active":"2023-11-08T08:20:33.037593+0000","last_became_peered":"2023-11-08T08:20:33.037593+0000","last_unstale":"2023-11-08T08:20:34.021093+0000","last_undegraded":"2023-11-08T08:20:34.021093+0000","last_fullsized":"2023-11-08T08:20:34.021093+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:00:34.853030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088368+0000","last_change":"2023-11-08T08:20:33.044048+0000","last_active":"2023-11-08T08:20:34.088368+0000","last_peered":"2023-11-08T08:20:34.088368+0000","last_clean":"2023-11-08T08:20:34.088368+0000","last_became_active":"2023-11-08T08:20:33.043735+0000","last_became_peered":"2023-11-08T08:20:33.043735+0000","last_unstale":"2023-11-08T08:20:34.088368+0000","last_undegraded":"2023-11-08T08:20:34.088368+0000","last_fullsized":"2023-11-08T08:20:34.088368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:36:37.828816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087559+0000","last_change":"2023-11-08T08:20:33.040511+0000","last_active":"2023-11-08T08:20:34.087559+0000","last_peered":"2023-11-08T08:20:34.087559+0000","last_clean":"2023-11-08T08:20:34.087559+0000","last_became_active":"2023-11-08T08:20:33.040226+0000","last_became_peered":"2023-11-08T08:20:33.040226+0000","last_unstale":"2023-11-08T08:20:34.087559+0000","last_undegraded":"2023-11-08T08:20:34.087559+0000","last_fullsized":"2023-11-08T08:20:34.087559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021092+0000","last_change":"2023-11-08T08:20:33.033215+0000","last_active":"2023-11-08T08:20:34.021092+0000","last_peered":"2023-11-08T08:20:34.021092+0000","last_clean":"2023-11-08T08:20:34.021092+0000","last_became_active":"2023-11-08T08:20:33.032935+0000","last_became_peered":"2023-11-08T08:20:33.032935+0000","last_unstale":"2023-11-08T08:20:34.021092+0000","last_undegraded":"2023-11-08T08:20:34.021092+0000","last_fullsized":"2023-11-08T08:20:34.021092+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:57:01.855036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"43'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088048+0000","last_change":"2023-11-08T08:20:32.647788+0000","last_active":"2023-11-08T08:20:34.088048+0000","last_peered":"2023-11-08T08:20:34.088048+0000","last_clean":"2023-11-08T08:20:34.088048+0000","last_became_active":"2023-11-08T08:20:32.647474+0000","last_became_peered":"2023-11-08T08:20:32.647474+0000","last_unstale":"2023-11-08T08:20:34.088048+0000","last_undegraded":"2023-11-08T08:20:34.088048+0000","last_fullsized":"2023-11-08T08:20:34.088048+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:36:36.511657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021064+0000","last_change":"2023-11-08T08:20:33.035667+0000","last_active":"2023-11-08T08:20:34.021064+0000","last_peered":"2023-11-08T08:20:34.021064+0000","last_clean":"2023-11-08T08:20:34.021064+0000","last_became_active":"2023-11-08T08:20:33.034101+0000","last_became_peered":"2023-11-08T08:20:33.034101+0000","last_unstale":"2023-11-08T08:20:34.021064+0000","last_undegraded":"2023-11-08T08:20:34.021064+0000","last_fullsized":"2023-11-08T08:20:34.021064+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:43:40.912492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022649+0000","last_change":"2023-11-08T08:20:32.649400+0000","last_active":"2023-11-08T08:20:34.022649+0000","last_peered":"2023-11-08T08:20:34.022649+0000","last_clean":"2023-11-08T08:20:34.022649+0000","last_became_active":"2023-11-08T08:20:32.649191+0000","last_became_peered":"2023-11-08T08:20:32.649191+0000","last_unstale":"2023-11-08T08:20:34.022649+0000","last_undegraded":"2023-11-08T08:20:34.022649+0000","last_fullsized":"2023-11-08T08:20:34.022649+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:55.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021207+0000","last_change":"2023-11-08T08:20:33.035274+0000","last_active":"2023-11-08T08:20:34.021207+0000","last_peered":"2023-11-08T08:20:34.021207+0000","last_clean":"2023-11-08T08:20:34.021207+0000","last_became_active":"2023-11-08T08:20:33.033839+0000","last_became_peered":"2023-11-08T08:20:33.033839+0000","last_unstale":"2023-11-08T08:20:34.021207+0000","last_undegraded":"2023-11-08T08:20:34.021207+0000","last_fullsized":"2023-11-08T08:20:34.021207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:44.931729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022141+0000","last_change":"2023-11-08T08:20:32.641284+0000","last_active":"2023-11-08T08:20:34.022141+0000","last_peered":"2023-11-08T08:20:34.022141+0000","last_clean":"2023-11-08T08:20:34.022141+0000","last_became_active":"2023-11-08T08:20:32.641014+0000","last_became_peered":"2023-11-08T08:20:32.641014+0000","last_unstale":"2023-11-08T08:20:34.022141+0000","last_undegraded":"2023-11-08T08:20:34.022141+0000","last_fullsized":"2023-11-08T08:20:34.022141+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:16.616131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021673+0000","last_change":"2023-11-08T08:20:33.036893+0000","last_active":"2023-11-08T08:20:34.021673+0000","last_peered":"2023-11-08T08:20:34.021673+0000","last_clean":"2023-11-08T08:20:34.021673+0000","last_became_active":"2023-11-08T08:20:33.036512+0000","last_became_peered":"2023-11-08T08:20:33.036512+0000","last_unstale":"2023-11-08T08:20:34.021673+0000","last_undegraded":"2023-11-08T08:20:34.021673+0000","last_fullsized":"2023-11-08T08:20:34.021673+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087904+0000","last_change":"2023-11-08T08:20:32.645234+0000","last_active":"2023-11-08T08:20:34.087904+0000","last_peered":"2023-11-08T08:20:34.087904+0000","last_clean":"2023-11-08T08:20:34.087904+0000","last_became_active":"2023-11-08T08:20:32.644949+0000","last_became_peered":"2023-11-08T08:20:32.644949+0000","last_unstale":"2023-11-08T08:20:34.087904+0000","last_undegraded":"2023-11-08T08:20:34.087904+0000","last_fullsized":"2023-11-08T08:20:34.087904+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:55.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087858+0000","last_change":"2023-11-08T08:20:33.025926+0000","last_active":"2023-11-08T08:20:34.087858+0000","last_peered":"2023-11-08T08:20:34.087858+0000","last_clean":"2023-11-08T08:20:34.087858+0000","last_became_active":"2023-11-08T08:20:33.025682+0000","last_became_peered":"2023-11-08T08:20:33.025682+0000","last_unstale":"2023-11-08T08:20:34.087858+0000","last_undegraded":"2023-11-08T08:20:34.087858+0000","last_fullsized":"2023-11-08T08:20:34.087858+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:56:33.178296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088310+0000","last_change":"2023-11-08T08:20:32.642448+0000","last_active":"2023-11-08T08:20:34.088310+0000","last_peered":"2023-11-08T08:20:34.088310+0000","last_clean":"2023-11-08T08:20:34.088310+0000","last_became_active":"2023-11-08T08:20:32.642135+0000","last_became_peered":"2023-11-08T08:20:32.642135+0000","last_unstale":"2023-11-08T08:20:34.088310+0000","last_undegraded":"2023-11-08T08:20:34.088310+0000","last_fullsized":"2023-11-08T08:20:34.088310+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:45.491146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021192+0000","last_change":"2023-11-08T08:20:33.035308+0000","last_active":"2023-11-08T08:20:34.021192+0000","last_peered":"2023-11-08T08:20:34.021192+0000","last_clean":"2023-11-08T08:20:34.021192+0000","last_became_active":"2023-11-08T08:20:33.035040+0000","last_became_peered":"2023-11-08T08:20:33.035040+0000","last_unstale":"2023-11-08T08:20:34.021192+0000","last_undegraded":"2023-11-08T08:20:34.021192+0000","last_fullsized":"2023-11-08T08:20:34.021192+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:13:11.268187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021304+0000","last_change":"2023-11-08T08:20:32.638835+0000","last_active":"2023-11-08T08:20:34.021304+0000","last_peered":"2023-11-08T08:20:34.021304+0000","last_clean":"2023-11-08T08:20:34.021304+0000","last_became_active":"2023-11-08T08:20:32.638553+0000","last_became_peered":"2023-11-08T08:20:32.638553+0000","last_unstale":"2023-11-08T08:20:34.021304+0000","last_undegraded":"2023-11-08T08:20:34.021304+0000","last_fullsized":"2023-11-08T08:20:34.021304+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:42:18.908130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022126+0000","last_change":"2023-11-08T08:20:33.037865+0000","last_active":"2023-11-08T08:20:34.022126+0000","last_peered":"2023-11-08T08:20:34.022126+0000","last_clean":"2023-11-08T08:20:34.022126+0000","last_became_active":"2023-11-08T08:20:33.037154+0000","last_became_peered":"2023-11-08T08:20:33.037154+0000","last_unstale":"2023-11-08T08:20:34.022126+0000","last_undegraded":"2023-11-08T08:20:34.022126+0000","last_fullsized":"2023-11-08T08:20:34.022126+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021051+0000","last_change":"2023-11-08T08:20:32.643968+0000","last_active":"2023-11-08T08:20:34.021051+0000","last_peered":"2023-11-08T08:20:34.021051+0000","last_clean":"2023-11-08T08:20:34.021051+0000","last_became_active":"2023-11-08T08:20:32.643610+0000","last_became_peered":"2023-11-08T08:20:32.643610+0000","last_unstale":"2023-11-08T08:20:34.021051+0000","last_undegraded":"2023-11-08T08:20:34.021051+0000","last_fullsized":"2023-11-08T08:20:34.021051+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:46:35.737983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088484+0000","last_change":"2023-11-08T08:20:33.042769+0000","last_active":"2023-11-08T08:20:34.088484+0000","last_peered":"2023-11-08T08:20:34.088484+0000","last_clean":"2023-11-08T08:20:34.088484+0000","last_became_active":"2023-11-08T08:20:33.042546+0000","last_became_peered":"2023-11-08T08:20:33.042546+0000","last_unstale":"2023-11-08T08:20:34.088484+0000","last_undegraded":"2023-11-08T08:20:34.088484+0000","last_fullsized":"2023-11-08T08:20:34.088484+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:42:19.915667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088077+0000","last_change":"2023-11-08T08:20:32.642541+0000","last_active":"2023-11-08T08:20:34.088077+0000","last_peered":"2023-11-08T08:20:34.088077+0000","last_clean":"2023-11-08T08:20:34.088077+0000","last_became_active":"2023-11-08T08:20:32.641890+0000","last_became_peered":"2023-11-08T08:20:32.641890+0000","last_unstale":"2023-11-08T08:20:34.088077+0000","last_undegraded":"2023-11-08T08:20:34.088077+0000","last_fullsized":"2023-11-08T08:20:34.088077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:21:08.720995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087402+0000","last_change":"2023-11-08T08:20:33.029212+0000","last_active":"2023-11-08T08:20:34.087402+0000","last_peered":"2023-11-08T08:20:34.087402+0000","last_clean":"2023-11-08T08:20:34.087402+0000","last_became_active":"2023-11-08T08:20:33.028931+0000","last_became_peered":"2023-11-08T08:20:33.028931+0000","last_unstale":"2023-11-08T08:20:34.087402+0000","last_undegraded":"2023-11-08T08:20:34.087402+0000","last_fullsized":"2023-11-08T08:20:34.087402+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088540+0000","last_change":"2023-11-08T08:20:32.638341+0000","last_active":"2023-11-08T08:20:34.088540+0000","last_peered":"2023-11-08T08:20:34.088540+0000","last_clean":"2023-11-08T08:20:34.088540+0000","last_became_active":"2023-11-08T08:20:32.638066+0000","last_became_peered":"2023-11-08T08:20:32.638066+0000","last_unstale":"2023-11-08T08:20:34.088540+0000","last_undegraded":"2023-11-08T08:20:34.088540+0000","last_fullsized":"2023-11-08T08:20:34.088540+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:38.025405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088361+0000","last_change":"2023-11-08T08:20:33.034928+0000","last_active":"2023-11-08T08:20:34.088361+0000","last_peered":"2023-11-08T08:20:34.088361+0000","last_clean":"2023-11-08T08:20:34.088361+0000","last_became_active":"2023-11-08T08:20:33.034671+0000","last_became_peered":"2023-11-08T08:20:33.034671+0000","last_unstale":"2023-11-08T08:20:34.088361+0000","last_undegraded":"2023-11-08T08:20:34.088361+0000","last_fullsized":"2023-11-08T08:20:34.088361+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:21:09.728532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088577+0000","last_change":"2023-11-08T08:20:32.645389+0000","last_active":"2023-11-08T08:20:34.088577+0000","last_peered":"2023-11-08T08:20:34.088577+0000","last_clean":"2023-11-08T08:20:34.088577+0000","last_became_active":"2023-11-08T08:20:32.645090+0000","last_became_peered":"2023-11-08T08:20:32.645090+0000","last_unstale":"2023-11-08T08:20:34.088577+0000","last_undegraded":"2023-11-08T08:20:34.088577+0000","last_fullsized":"2023-11-08T08:20:34.088577+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:19.636821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088582+0000","last_change":"2023-11-08T08:20:33.037707+0000","last_active":"2023-11-08T08:20:34.088582+0000","last_peered":"2023-11-08T08:20:34.088582+0000","last_clean":"2023-11-08T08:20:34.088582+0000","last_became_active":"2023-11-08T08:20:33.037444+0000","last_became_peered":"2023-11-08T08:20:33.037444+0000","last_unstale":"2023-11-08T08:20:34.088582+0000","last_undegraded":"2023-11-08T08:20:34.088582+0000","last_fullsized":"2023-11-08T08:20:34.088582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"43'3","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021292+0000","last_change":"2023-11-08T08:20:32.641224+0000","last_active":"2023-11-08T08:20:34.021292+0000","last_peered":"2023-11-08T08:20:34.021292+0000","last_clean":"2023-11-08T08:20:34.021292+0000","last_became_active":"2023-11-08T08:20:32.640873+0000","last_became_peered":"2023-11-08T08:20:32.640873+0000","last_unstale":"2023-11-08T08:20:34.021292+0000","last_undegraded":"2023-11-08T08:20:34.021292+0000","last_fullsized":"2023-11-08T08:20:34.021292+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:07:00.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087887+0000","last_change":"2023-11-08T08:20:33.026829+0000","last_active":"2023-11-08T08:20:34.087887+0000","last_peered":"2023-11-08T08:20:34.087887+0000","last_clean":"2023-11-08T08:20:34.087887+0000","last_became_active":"2023-11-08T08:20:33.026569+0000","last_became_peered":"2023-11-08T08:20:33.026569+0000","last_unstale":"2023-11-08T08:20:34.087887+0000","last_undegraded":"2023-11-08T08:20:34.087887+0000","last_fullsized":"2023-11-08T08:20:34.087887+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:42:30.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"43'6","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022169+0000","last_change":"2023-11-08T08:20:32.646660+0000","last_active":"2023-11-08T08:20:34.022169+0000","last_peered":"2023-11-08T08:20:34.022169+0000","last_clean":"2023-11-08T08:20:34.022169+0000","last_became_active":"2023-11-08T08:20:32.645049+0000","last_became_peered":"2023-11-08T08:20:32.645049+0000","last_unstale":"2023-11-08T08:20:34.022169+0000","last_undegraded":"2023-11-08T08:20:34.022169+0000","last_fullsized":"2023-11-08T08:20:34.022169+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:14:18.361427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021338+0000","last_change":"2023-11-08T08:20:33.035535+0000","last_active":"2023-11-08T08:20:34.021338+0000","last_peered":"2023-11-08T08:20:34.021338+0000","last_clean":"2023-11-08T08:20:34.021338+0000","last_became_active":"2023-11-08T08:20:33.034052+0000","last_became_peered":"2023-11-08T08:20:33.034052+0000","last_unstale":"2023-11-08T08:20:34.021338+0000","last_undegraded":"2023-11-08T08:20:34.021338+0000","last_fullsized":"2023-11-08T08:20:34.021338+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:04:10.931634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021716+0000","last_change":"2023-11-08T08:20:32.646559+0000","last_active":"2023-11-08T08:20:34.021716+0000","last_peered":"2023-11-08T08:20:34.021716+0000","last_clean":"2023-11-08T08:20:34.021716+0000","last_became_active":"2023-11-08T08:20:32.646258+0000","last_became_peered":"2023-11-08T08:20:32.646258+0000","last_unstale":"2023-11-08T08:20:34.021716+0000","last_undegraded":"2023-11-08T08:20:34.021716+0000","last_fullsized":"2023-11-08T08:20:34.021716+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:43:45.224233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088899+0000","last_change":"2023-11-08T08:20:33.046636+0000","last_active":"2023-11-08T08:20:34.088899+0000","last_peered":"2023-11-08T08:20:34.088899+0000","last_clean":"2023-11-08T08:20:34.088899+0000","last_became_active":"2023-11-08T08:20:33.046400+0000","last_became_peered":"2023-11-08T08:20:33.046400+0000","last_unstale":"2023-11-08T08:20:34.088899+0000","last_undegraded":"2023-11-08T08:20:34.088899+0000","last_fullsized":"2023-11-08T08:20:34.088899+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:51:02.264321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088386+0000","last_change":"2023-11-08T08:20:32.642090+0000","last_active":"2023-11-08T08:20:34.088386+0000","last_peered":"2023-11-08T08:20:34.088386+0000","last_clean":"2023-11-08T08:20:34.088386+0000","last_became_active":"2023-11-08T08:20:32.641772+0000","last_became_peered":"2023-11-08T08:20:32.641772+0000","last_unstale":"2023-11-08T08:20:34.088386+0000","last_undegraded":"2023-11-08T08:20:34.088386+0000","last_fullsized":"2023-11-08T08:20:34.088386+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:22.535882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022530+0000","last_change":"2023-11-08T08:20:33.039967+0000","last_active":"2023-11-08T08:20:34.022530+0000","last_peered":"2023-11-08T08:20:34.022530+0000","last_clean":"2023-11-08T08:20:34.022530+0000","last_became_active":"2023-11-08T08:20:33.039641+0000","last_became_peered":"2023-11-08T08:20:33.039641+0000","last_unstale":"2023-11-08T08:20:34.022530+0000","last_undegraded":"2023-11-08T08:20:34.022530+0000","last_fullsized":"2023-11-08T08:20:34.022530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087779+0000","last_change":"2023-11-08T08:20:32.641027+0000","last_active":"2023-11-08T08:20:34.087779+0000","last_peered":"2023-11-08T08:20:34.087779+0000","last_clean":"2023-11-08T08:20:34.087779+0000","last_became_active":"2023-11-08T08:20:32.640691+0000","last_became_peered":"2023-11-08T08:20:32.640691+0000","last_unstale":"2023-11-08T08:20:34.087779+0000","last_undegraded":"2023-11-08T08:20:34.087779+0000","last_fullsized":"2023-11-08T08:20:34.087779+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:43:45.224233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088908+0000","last_change":"2023-11-08T08:20:33.034299+0000","last_active":"2023-11-08T08:20:34.088908+0000","last_peered":"2023-11-08T08:20:34.088908+0000","last_clean":"2023-11-08T08:20:34.088908+0000","last_became_active":"2023-11-08T08:20:33.033821+0000","last_became_peered":"2023-11-08T08:20:33.033821+0000","last_unstale":"2023-11-08T08:20:34.088908+0000","last_undegraded":"2023-11-08T08:20:34.088908+0000","last_fullsized":"2023-11-08T08:20:34.088908+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:43:15.255479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"43'8","reported_seq":19,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021270+0000","last_change":"2023-11-08T08:20:32.647360+0000","last_active":"2023-11-08T08:20:34.021270+0000","last_peered":"2023-11-08T08:20:34.021270+0000","last_clean":"2023-11-08T08:20:34.021270+0000","last_became_active":"2023-11-08T08:20:32.647033+0000","last_became_peered":"2023-11-08T08:20:32.647033+0000","last_unstale":"2023-11-08T08:20:34.021270+0000","last_undegraded":"2023-11-08T08:20:34.021270+0000","last_fullsized":"2023-11-08T08:20:34.021270+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:55:30.574115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021582+0000","last_change":"2023-11-08T08:20:33.029834+0000","last_active":"2023-11-08T08:20:34.021582+0000","last_peered":"2023-11-08T08:20:34.021582+0000","last_clean":"2023-11-08T08:20:34.021582+0000","last_became_active":"2023-11-08T08:20:33.029598+0000","last_became_peered":"2023-11-08T08:20:33.029598+0000","last_unstale":"2023-11-08T08:20:34.021582+0000","last_undegraded":"2023-11-08T08:20:34.021582+0000","last_fullsized":"2023-11-08T08:20:34.021582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022722+0000","last_change":"2023-11-08T08:20:32.649522+0000","last_active":"2023-11-08T08:20:34.022722+0000","last_peered":"2023-11-08T08:20:34.022722+0000","last_clean":"2023-11-08T08:20:34.022722+0000","last_became_active":"2023-11-08T08:20:32.649239+0000","last_became_peered":"2023-11-08T08:20:32.649239+0000","last_unstale":"2023-11-08T08:20:34.022722+0000","last_undegraded":"2023-11-08T08:20:34.022722+0000","last_fullsized":"2023-11-08T08:20:34.022722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:14:18.361427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088106+0000","last_change":"2023-11-08T08:20:33.038087+0000","last_active":"2023-11-08T08:20:34.088106+0000","last_peered":"2023-11-08T08:20:34.088106+0000","last_clean":"2023-11-08T08:20:34.088106+0000","last_became_active":"2023-11-08T08:20:33.037806+0000","last_became_peered":"2023-11-08T08:20:33.037806+0000","last_unstale":"2023-11-08T08:20:34.088106+0000","last_undegraded":"2023-11-08T08:20:34.088106+0000","last_fullsized":"2023-11-08T08:20:34.088106+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:23.543419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022638+0000","last_change":"2023-11-08T08:20:32.649879+0000","last_active":"2023-11-08T08:20:34.022638+0000","last_peered":"2023-11-08T08:20:34.022638+0000","last_clean":"2023-11-08T08:20:34.022638+0000","last_became_active":"2023-11-08T08:20:32.649619+0000","last_became_peered":"2023-11-08T08:20:32.649619+0000","last_unstale":"2023-11-08T08:20:34.022638+0000","last_undegraded":"2023-11-08T08:20:34.022638+0000","last_fullsized":"2023-11-08T08:20:34.022638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:47:24.700788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087562+0000","last_change":"2023-11-08T08:20:33.027549+0000","last_active":"2023-11-08T08:20:34.087562+0000","last_peered":"2023-11-08T08:20:34.087562+0000","last_clean":"2023-11-08T08:20:34.087562+0000","last_became_active":"2023-11-08T08:20:33.027312+0000","last_became_peered":"2023-11-08T08:20:33.027312+0000","last_unstale":"2023-11-08T08:20:34.087562+0000","last_undegraded":"2023-11-08T08:20:34.087562+0000","last_fullsized":"2023-11-08T08:20:34.087562+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088652+0000","last_change":"2023-11-08T08:20:32.642896+0000","last_active":"2023-11-08T08:20:34.088652+0000","last_peered":"2023-11-08T08:20:34.088652+0000","last_clean":"2023-11-08T08:20:34.088652+0000","last_became_active":"2023-11-08T08:20:32.642674+0000","last_became_peered":"2023-11-08T08:20:32.642674+0000","last_unstale":"2023-11-08T08:20:34.088652+0000","last_undegraded":"2023-11-08T08:20:34.088652+0000","last_fullsized":"2023-11-08T08:20:34.088652+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:21.875398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021609+0000","last_change":"2023-11-08T08:20:33.047525+0000","last_active":"2023-11-08T08:20:34.021609+0000","last_peered":"2023-11-08T08:20:34.021609+0000","last_clean":"2023-11-08T08:20:34.021609+0000","last_became_active":"2023-11-08T08:20:33.047223+0000","last_became_peered":"2023-11-08T08:20:33.047223+0000","last_unstale":"2023-11-08T08:20:34.021609+0000","last_undegraded":"2023-11-08T08:20:34.021609+0000","last_fullsized":"2023-11-08T08:20:34.021609+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:42:30.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087763+0000","last_change":"2023-11-08T08:20:32.641498+0000","last_active":"2023-11-08T08:20:34.087763+0000","last_peered":"2023-11-08T08:20:34.087763+0000","last_clean":"2023-11-08T08:20:34.087763+0000","last_became_active":"2023-11-08T08:20:32.641273+0000","last_became_peered":"2023-11-08T08:20:32.641273+0000","last_unstale":"2023-11-08T08:20:34.087763+0000","last_undegraded":"2023-11-08T08:20:34.087763+0000","last_fullsized":"2023-11-08T08:20:34.087763+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:55:30.574115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087865+0000","last_change":"2023-11-08T08:20:33.031258+0000","last_active":"2023-11-08T08:20:34.087865+0000","last_peered":"2023-11-08T08:20:34.087865+0000","last_clean":"2023-11-08T08:20:34.087865+0000","last_became_active":"2023-11-08T08:20:33.031004+0000","last_became_peered":"2023-11-08T08:20:33.031004+0000","last_unstale":"2023-11-08T08:20:34.087865+0000","last_undegraded":"2023-11-08T08:20:34.087865+0000","last_fullsized":"2023-11-08T08:20:34.087865+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088409+0000","last_change":"2023-11-08T08:20:32.645648+0000","last_active":"2023-11-08T08:20:34.088409+0000","last_peered":"2023-11-08T08:20:34.088409+0000","last_clean":"2023-11-08T08:20:34.088409+0000","last_became_active":"2023-11-08T08:20:32.645357+0000","last_became_peered":"2023-11-08T08:20:32.645357+0000","last_unstale":"2023-11-08T08:20:34.088409+0000","last_undegraded":"2023-11-08T08:20:34.088409+0000","last_fullsized":"2023-11-08T08:20:34.088409+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:16.616131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088566+0000","last_change":"2023-11-08T08:20:33.035942+0000","last_active":"2023-11-08T08:20:34.088566+0000","last_peered":"2023-11-08T08:20:34.088566+0000","last_clean":"2023-11-08T08:20:34.088566+0000","last_became_active":"2023-11-08T08:20:33.035195+0000","last_became_peered":"2023-11-08T08:20:33.035195+0000","last_unstale":"2023-11-08T08:20:34.088566+0000","last_undegraded":"2023-11-08T08:20:34.088566+0000","last_fullsized":"2023-11-08T08:20:34.088566+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:11:41.599485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088789+0000","last_change":"2023-11-08T08:20:32.644955+0000","last_active":"2023-11-08T08:20:34.088789+0000","last_peered":"2023-11-08T08:20:34.088789+0000","last_clean":"2023-11-08T08:20:34.088789+0000","last_became_active":"2023-11-08T08:20:32.644690+0000","last_became_peered":"2023-11-08T08:20:32.644690+0000","last_unstale":"2023-11-08T08:20:34.088789+0000","last_undegraded":"2023-11-08T08:20:34.088789+0000","last_fullsized":"2023-11-08T08:20:34.088789+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:42:18.908130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088379+0000","last_change":"2023-11-08T08:20:33.037993+0000","last_active":"2023-11-08T08:20:34.088379+0000","last_peered":"2023-11-08T08:20:34.088379+0000","last_clean":"2023-11-08T08:20:34.088379+0000","last_became_active":"2023-11-08T08:20:33.037173+0000","last_became_peered":"2023-11-08T08:20:33.037173+0000","last_unstale":"2023-11-08T08:20:34.088379+0000","last_undegraded":"2023-11-08T08:20:34.088379+0000","last_fullsized":"2023-11-08T08:20:34.088379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:08:02.296246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021822+0000","last_change":"2023-11-08T08:20:32.644960+0000","last_active":"2023-11-08T08:20:34.021822+0000","last_peered":"2023-11-08T08:20:34.021822+0000","last_clean":"2023-11-08T08:20:34.021822+0000","last_became_active":"2023-11-08T08:20:32.642226+0000","last_became_peered":"2023-11-08T08:20:32.642226+0000","last_unstale":"2023-11-08T08:20:34.021822+0000","last_undegraded":"2023-11-08T08:20:34.021822+0000","last_fullsized":"2023-11-08T08:20:34.021822+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:55.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022664+0000","last_change":"2023-11-08T08:20:33.044543+0000","last_active":"2023-11-08T08:20:34.022664+0000","last_peered":"2023-11-08T08:20:34.022664+0000","last_clean":"2023-11-08T08:20:34.022664+0000","last_became_active":"2023-11-08T08:20:33.044282+0000","last_became_peered":"2023-11-08T08:20:33.044282+0000","last_unstale":"2023-11-08T08:20:34.022664+0000","last_undegraded":"2023-11-08T08:20:34.022664+0000","last_fullsized":"2023-11-08T08:20:34.022664+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:23.543419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021581+0000","last_change":"2023-11-08T08:20:32.644350+0000","last_active":"2023-11-08T08:20:34.021581+0000","last_peered":"2023-11-08T08:20:34.021581+0000","last_clean":"2023-11-08T08:20:34.021581+0000","last_became_active":"2023-11-08T08:20:32.641483+0000","last_became_peered":"2023-11-08T08:20:32.641483+0000","last_unstale":"2023-11-08T08:20:34.021581+0000","last_undegraded":"2023-11-08T08:20:34.021581+0000","last_fullsized":"2023-11-08T08:20:34.021581+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:31:18.529121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022029+0000","last_change":"2023-11-08T08:20:33.039195+0000","last_active":"2023-11-08T08:20:34.022029+0000","last_peered":"2023-11-08T08:20:34.022029+0000","last_clean":"2023-11-08T08:20:34.022029+0000","last_became_active":"2023-11-08T08:20:33.038954+0000","last_became_peered":"2023-11-08T08:20:33.038954+0000","last_unstale":"2023-11-08T08:20:34.022029+0000","last_undegraded":"2023-11-08T08:20:34.022029+0000","last_fullsized":"2023-11-08T08:20:34.022029+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021729+0000","last_change":"2023-11-08T08:20:32.640819+0000","last_active":"2023-11-08T08:20:34.021729+0000","last_peered":"2023-11-08T08:20:34.021729+0000","last_clean":"2023-11-08T08:20:34.021729+0000","last_became_active":"2023-11-08T08:20:32.640305+0000","last_became_peered":"2023-11-08T08:20:32.640305+0000","last_unstale":"2023-11-08T08:20:34.021729+0000","last_undegraded":"2023-11-08T08:20:34.021729+0000","last_fullsized":"2023-11-08T08:20:34.021729+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:55:30.574115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.016572+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:20:33.016572+0000","last_peered":"2023-11-08T08:20:33.016572+0000","last_clean":"2023-11-08T08:20:33.016572+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:20:33.016572+0000","last_undegraded":"2023-11-08T08:20:33.016572+0000","last_fullsized":"2023-11-08T08:20:33.016572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088228+0000","last_change":"2023-11-08T08:20:33.027420+0000","last_active":"2023-11-08T08:20:34.088228+0000","last_peered":"2023-11-08T08:20:34.088228+0000","last_clean":"2023-11-08T08:20:34.088228+0000","last_became_active":"2023-11-08T08:20:33.027182+0000","last_became_peered":"2023-11-08T08:20:33.027182+0000","last_unstale":"2023-11-08T08:20:34.088228+0000","last_undegraded":"2023-11-08T08:20:34.088228+0000","last_fullsized":"2023-11-08T08:20:34.088228+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088902+0000","last_change":"2023-11-08T08:20:32.645392+0000","last_active":"2023-11-08T08:20:34.088902+0000","last_peered":"2023-11-08T08:20:34.088902+0000","last_clean":"2023-11-08T08:20:34.088902+0000","last_became_active":"2023-11-08T08:20:32.644516+0000","last_became_peered":"2023-11-08T08:20:32.644516+0000","last_unstale":"2023-11-08T08:20:34.088902+0000","last_undegraded":"2023-11-08T08:20:34.088902+0000","last_fullsized":"2023-11-08T08:20:34.088902+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:07:00.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.029305+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:20:33.029305+0000","last_peered":"2023-11-08T08:20:33.029305+0000","last_clean":"2023-11-08T08:20:33.029305+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:20:33.029305+0000","last_undegraded":"2023-11-08T08:20:33.029305+0000","last_fullsized":"2023-11-08T08:20:33.029305+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022217+0000","last_change":"2023-11-08T08:20:33.036789+0000","last_active":"2023-11-08T08:20:34.022217+0000","last_peered":"2023-11-08T08:20:34.022217+0000","last_clean":"2023-11-08T08:20:34.022217+0000","last_became_active":"2023-11-08T08:20:33.036498+0000","last_became_peered":"2023-11-08T08:20:33.036498+0000","last_unstale":"2023-11-08T08:20:34.022217+0000","last_undegraded":"2023-11-08T08:20:34.022217+0000","last_fullsized":"2023-11-08T08:20:34.022217+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:08:02.296246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021335+0000","last_change":"2023-11-08T08:20:32.641457+0000","last_active":"2023-11-08T08:20:34.021335+0000","last_peered":"2023-11-08T08:20:34.021335+0000","last_clean":"2023-11-08T08:20:34.021335+0000","last_became_active":"2023-11-08T08:20:32.641214+0000","last_became_peered":"2023-11-08T08:20:32.641214+0000","last_unstale":"2023-11-08T08:20:34.021335+0000","last_undegraded":"2023-11-08T08:20:34.021335+0000","last_fullsized":"2023-11-08T08:20:34.021335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:45.491146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.025713+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:20:33.025713+0000","last_peered":"2023-11-08T08:20:33.025713+0000","last_clean":"2023-11-08T08:20:33.025713+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:20:33.025713+0000","last_undegraded":"2023-11-08T08:20:33.025713+0000","last_fullsized":"2023-11-08T08:20:33.025713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":187,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.023225+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:20:33.023225+0000","last_peered":"2023-11-08T08:20:33.023225+0000","last_clean":"2023-11-08T08:20:33.023225+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:20:33.023225+0000","last_undegraded":"2023-11-08T08:20:33.023225+0000","last_fullsized":"2023-11-08T08:20:33.023225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088423+0000","last_change":"2023-11-08T08:20:33.035325+0000","last_active":"2023-11-08T08:20:34.088423+0000","last_peered":"2023-11-08T08:20:34.088423+0000","last_clean":"2023-11-08T08:20:34.088423+0000","last_became_active":"2023-11-08T08:20:33.035014+0000","last_became_peered":"2023-11-08T08:20:33.035014+0000","last_unstale":"2023-11-08T08:20:34.088423+0000","last_undegraded":"2023-11-08T08:20:34.088423+0000","last_fullsized":"2023-11-08T08:20:34.088423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:00:34.853030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088139+0000","last_change":"2023-11-08T08:20:32.644969+0000","last_active":"2023-11-08T08:20:34.088139+0000","last_peered":"2023-11-08T08:20:34.088139+0000","last_clean":"2023-11-08T08:20:34.088139+0000","last_became_active":"2023-11-08T08:20:32.644727+0000","last_became_peered":"2023-11-08T08:20:32.644727+0000","last_unstale":"2023-11-08T08:20:34.088139+0000","last_undegraded":"2023-11-08T08:20:34.088139+0000","last_fullsized":"2023-11-08T08:20:34.088139+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:46:35.737983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.020635+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:20:33.020635+0000","last_peered":"2023-11-08T08:20:33.020635+0000","last_clean":"2023-11-08T08:20:33.020635+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:20:33.020635+0000","last_undegraded":"2023-11-08T08:20:33.020635+0000","last_fullsized":"2023-11-08T08:20:33.020635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087469+0000","last_change":"2023-11-08T08:20:33.026312+0000","last_active":"2023-11-08T08:20:34.087469+0000","last_peered":"2023-11-08T08:20:34.087469+0000","last_clean":"2023-11-08T08:20:34.087469+0000","last_became_active":"2023-11-08T08:20:33.026055+0000","last_became_peered":"2023-11-08T08:20:33.026055+0000","last_unstale":"2023-11-08T08:20:34.087469+0000","last_undegraded":"2023-11-08T08:20:34.087469+0000","last_fullsized":"2023-11-08T08:20:34.087469+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:08:14.386212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022262+0000","last_change":"2023-11-08T08:20:32.640456+0000","last_active":"2023-11-08T08:20:34.022262+0000","last_peered":"2023-11-08T08:20:34.022262+0000","last_clean":"2023-11-08T08:20:34.022262+0000","last_became_active":"2023-11-08T08:20:32.640147+0000","last_became_peered":"2023-11-08T08:20:32.640147+0000","last_unstale":"2023-11-08T08:20:34.022262+0000","last_undegraded":"2023-11-08T08:20:34.022262+0000","last_fullsized":"2023-11-08T08:20:34.022262+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:27:43.728859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.028736+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:20:33.028736+0000","last_peered":"2023-11-08T08:20:33.028736+0000","last_clean":"2023-11-08T08:20:33.028736+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:20:33.028736+0000","last_undegraded":"2023-11-08T08:20:33.028736+0000","last_fullsized":"2023-11-08T08:20:33.028736+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088252+0000","last_change":"2023-11-08T08:20:33.046426+0000","last_active":"2023-11-08T08:20:34.088252+0000","last_peered":"2023-11-08T08:20:34.088252+0000","last_clean":"2023-11-08T08:20:34.088252+0000","last_became_active":"2023-11-08T08:20:33.046206+0000","last_became_peered":"2023-11-08T08:20:33.046206+0000","last_unstale":"2023-11-08T08:20:34.088252+0000","last_undegraded":"2023-11-08T08:20:34.088252+0000","last_fullsized":"2023-11-08T08:20:34.088252+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:46.498683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021181+0000","last_change":"2023-11-08T08:20:32.644433+0000","last_active":"2023-11-08T08:20:34.021181+0000","last_peered":"2023-11-08T08:20:34.021181+0000","last_clean":"2023-11-08T08:20:34.021181+0000","last_became_active":"2023-11-08T08:20:32.644229+0000","last_became_peered":"2023-11-08T08:20:32.644229+0000","last_unstale":"2023-11-08T08:20:34.021181+0000","last_undegraded":"2023-11-08T08:20:34.021181+0000","last_fullsized":"2023-11-08T08:20:34.021181+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:21:08.720995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.023324+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:20:33.023324+0000","last_peered":"2023-11-08T08:20:33.023324+0000","last_clean":"2023-11-08T08:20:33.023324+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:20:33.023324+0000","last_undegraded":"2023-11-08T08:20:33.023324+0000","last_fullsized":"2023-11-08T08:20:33.023324+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022645+0000","last_change":"2023-11-08T08:20:33.042016+0000","last_active":"2023-11-08T08:20:34.022645+0000","last_peered":"2023-11-08T08:20:34.022645+0000","last_clean":"2023-11-08T08:20:34.022645+0000","last_became_active":"2023-11-08T08:20:33.040745+0000","last_became_peered":"2023-11-08T08:20:33.040745+0000","last_unstale":"2023-11-08T08:20:34.022645+0000","last_undegraded":"2023-11-08T08:20:34.022645+0000","last_fullsized":"2023-11-08T08:20:34.022645+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088093+0000","last_change":"2023-11-08T08:20:32.642179+0000","last_active":"2023-11-08T08:20:34.088093+0000","last_peered":"2023-11-08T08:20:34.088093+0000","last_clean":"2023-11-08T08:20:34.088093+0000","last_became_active":"2023-11-08T08:20:32.641860+0000","last_became_peered":"2023-11-08T08:20:32.641860+0000","last_unstale":"2023-11-08T08:20:34.088093+0000","last_undegraded":"2023-11-08T08:20:34.088093+0000","last_fullsized":"2023-11-08T08:20:34.088093+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:51:01.256784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.023048+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:20:33.023048+0000","last_peered":"2023-11-08T08:20:33.023048+0000","last_clean":"2023-11-08T08:20:33.023048+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:20:33.023048+0000","last_undegraded":"2023-11-08T08:20:33.023048+0000","last_fullsized":"2023-11-08T08:20:33.023048+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022265+0000","last_change":"2023-11-08T08:20:33.036733+0000","last_active":"2023-11-08T08:20:34.022265+0000","last_peered":"2023-11-08T08:20:34.022265+0000","last_clean":"2023-11-08T08:20:34.022265+0000","last_became_active":"2023-11-08T08:20:33.036448+0000","last_became_peered":"2023-11-08T08:20:33.036448+0000","last_unstale":"2023-11-08T08:20:34.022265+0000","last_undegraded":"2023-11-08T08:20:34.022265+0000","last_fullsized":"2023-11-08T08:20:34.022265+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:42:30.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087865+0000","last_change":"2023-11-08T08:20:32.637682+0000","last_active":"2023-11-08T08:20:34.087865+0000","last_peered":"2023-11-08T08:20:34.087865+0000","last_clean":"2023-11-08T08:20:34.087865+0000","last_became_active":"2023-11-08T08:20:32.637425+0000","last_became_peered":"2023-11-08T08:20:32.637425+0000","last_unstale":"2023-11-08T08:20:34.087865+0000","last_undegraded":"2023-11-08T08:20:34.087865+0000","last_fullsized":"2023-11-08T08:20:34.087865+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:14:18.361427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.030141+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:20:33.030141+0000","last_peered":"2023-11-08T08:20:33.030141+0000","last_clean":"2023-11-08T08:20:33.030141+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:20:33.030141+0000","last_undegraded":"2023-11-08T08:20:33.030141+0000","last_fullsized":"2023-11-08T08:20:33.030141+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8776,"kb_used_data":512,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363064,"statfs":{"total":96636764160,"available":96627777536,"internally_reserved":0,"allocated":524288,"data_stored":219994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.45600000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":5.8410000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.54100000000000004}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.624}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.69799999999999995}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66600000000000004}]},{"osd":6,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.65100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8832,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363008,"statfs":{"total":96636764160,"available":96627720192,"internally_reserved":0,"allocated":516096,"data_stored":219243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45800000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.433}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.51200000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.5},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.58599999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8900,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362940,"statfs":{"total":96636764160,"available":96627650560,"internally_reserved":0,"allocated":520192,"data_stored":220008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.252},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.309}]},{"osd":4,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.371}]},{"osd":5,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.33100000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42499999999999999}]},{"osd":7,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.36699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8836,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363004,"statfs":{"total":96636764160,"available":96627716096,"internally_reserved":0,"allocated":520192,"data_stored":220504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.434},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.47399999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.72299999999999998}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.628}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.65200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8900,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362940,"statfs":{"total":96636764160,"available":96627650560,"internally_reserved":0,"allocated":520192,"data_stored":219972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.314}]},{"osd":3,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"last":0.379}]},{"osd":4,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54200000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.621}]},{"osd":6,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66900000000000004}]},{"osd":7,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10056,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94361784,"statfs":{"total":96636764160,"available":96626466816,"internally_reserved":0,"allocated":1114112,"data_stored":810396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.5070000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.502},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.56599999999999995}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.876},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.76000000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.91500000000000004}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.85399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9108,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362732,"statfs":{"total":96636764160,"available":96627437568,"internally_reserved":0,"allocated":536576,"data_stored":221578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59799999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":1.3660000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.44},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":1.492}]},{"osd":5,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.64700000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53600000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9940,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94361900,"statfs":{"total":96636764160,"available":96626585600,"internally_reserved":0,"allocated":1126400,"data_stored":811944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.6790000000000003,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.7270000000000003,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.56000000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":7.6079999999999997,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.495},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.0209999999999999,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":9.7550000000000008,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.5489999999999999,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.36099999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.379},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45500000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.438},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:20:37.086 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:20:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.237+0000 7f293940b700 1 Processor -- start 2023-11-08T08:20:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.237+0000 7f293940b700 1 -- start start 2023-11-08T08:20:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f293940b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f293406b7b0 0x7f2934072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f293940b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f29340ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f293940b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340726c0 con 0x7f29340b0880 2023-11-08T08:20:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f293940b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293406bc50 con 0x7f293406b7b0 2023-11-08T08:20:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f293940b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29340b43f0 con 0x7f29340ad1f0 2023-11-08T08:20:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f29340ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29327fc700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f293406b7b0 0x7f2934072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f29340ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:40060/0 (socket says 172.21.15.116:40060) 2023-11-08T08:20:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 -- 172.21.15.116:0/2361947060 learned_addr learned my addr 172.21.15.116:0/2361947060 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f2932ffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f29340b0880 0x7f29340aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:39776/0 (socket says 172.21.15.116:39776) 2023-11-08T08:20:37.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 -- 172.21.15.116:0/2361947060 >> v2:172.21.15.129:3300/0 conn(0x7f293406b7b0 msgr2=0x7f2934072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:37.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 --2- 172.21.15.116:0/2361947060 >> v2:172.21.15.129:3300/0 conn(0x7f293406b7b0 0x7f2934072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 -- 172.21.15.116:0/2361947060 >> v1:172.21.15.129:6789/0 conn(0x7f29340b0880 legacy=0x7f29340aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29337fe700 1 -- 172.21.15.116:0/2361947060 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340adb90 con 0x7f29340ad1f0 2023-11-08T08:20:37.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.238+0000 7f29327fc700 1 --2- 172.21.15.116:0/2361947060 >> v2:172.21.15.129:3300/0 conn(0x7f293406b7b0 0x7f2934072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:37.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f29337fe700 1 --2- 172.21.15.116:0/2361947060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f29340ad650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f292c009e50 tx=0x7f292c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c3be2a96e50caf server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:37.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f2931ffb700 1 -- 172.21.15.116:0/2361947060 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c00e070 con 0x7f29340ad1f0 2023-11-08T08:20:37.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f2931ffb700 1 -- 172.21.15.116:0/2361947060 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c00b910 con 0x7f29340ad1f0 2023-11-08T08:20:37.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f2931ffb700 1 -- 172.21.15.116:0/2361947060 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c00bcc0 con 0x7f29340ad1f0 2023-11-08T08:20:37.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 -- 172.21.15.116:0/2361947060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 msgr2=0x7f29340ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 --2- 172.21.15.116:0/2361947060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f29340ad650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f292c009e50 tx=0x7f292c00b330 comp rx=0 tx=0).stop 2023-11-08T08:20:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 -- 172.21.15.116:0/2361947060 shutdown_connections 2023-11-08T08:20:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 --2- 172.21.15.116:0/2361947060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f29340ad650 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 --2- 172.21.15.116:0/2361947060 >> v2:172.21.15.129:3300/0 conn(0x7f293406b7b0 0x7f2934072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 -- 172.21.15.116:0/2361947060 >> 172.21.15.116:0/2361947060 conn(0x7f29340b2fc0 msgr2=0x7f2934071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:37.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 -- 172.21.15.116:0/2361947060 shutdown_connections 2023-11-08T08:20:37.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.239+0000 7f293940b700 1 -- 172.21.15.116:0/2361947060 wait complete. 2023-11-08T08:20:37.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 Processor -- start 2023-11-08T08:20:37.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 -- start start 2023-11-08T08:20:37.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f293406b7b0 0x7f293415b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f293415f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 0x7f293415fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293415e070 con 0x7f29340b0880 2023-11-08T08:20:37.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293415e350 con 0x7f293406b7b0 2023-11-08T08:20:37.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f293940b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293415e630 con 0x7f29340ad1f0 2023-11-08T08:20:37.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.241+0000 7f29337fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 0x7f293415fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 0x7f293415fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41964/0 (socket says 172.21.15.116:41964) 2023-11-08T08:20:37.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 -- 172.21.15.116:0/2077159860 learned_addr learned my addr 172.21.15.116:0/2077159860 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:37.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29327fc700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f293415f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f2932ffd700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f293406b7b0 0x7f293415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 -- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 msgr2=0x7f293415f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f293415f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 -- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f293406b7b0 msgr2=0x7f293415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f293406b7b0 0x7f293415b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 -- 172.21.15.116:0/2077159860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280094a0 con 0x7f29340b0880 2023-11-08T08:20:37.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f2932ffd700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f293406b7b0 0x7f293415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:37.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f29337fe700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 0x7f293415fb40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2934073d10 tx=0x7f292c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:37.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c0048b0 con 0x7f29340b0880 2023-11-08T08:20:37.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c0099b0 con 0x7f29340b0880 2023-11-08T08:20:37.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292c009b30 con 0x7f29340b0880 2023-11-08T08:20:37.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c01c680 con 0x7f29340b0880 2023-11-08T08:20:37.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.242+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29340b4d00 con 0x7f29340b0880 2023-11-08T08:20:37.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.243+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29340b46d0 con 0x7f29340b0880 2023-11-08T08:20:37.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.243+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f292c02c070 con 0x7f29340b0880 2023-11-08T08:20:37.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.244+0000 7f291b7fe700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29200682f0 0x7f292006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.244+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f292c087990 con 0x7f29340b0880 2023-11-08T08:20:37.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.244+0000 7f2932ffd700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29200682f0 0x7f292006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.244+0000 7f2932ffd700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29200682f0 0x7f292006a7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2924000f60 tx=0x7f2924007dd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:37.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.248+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f292c054080 con 0x7f29340b0880 2023-11-08T08:20:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.425+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f293415e4c0 con 0x7f29200682f0 2023-11-08T08:20:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.434+0000 7f291b7fe700 1 -- 172.21.15.116:0/2077159860 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311410 (secure 0 0 0) 0x7f293415e4c0 con 0x7f29200682f0 2023-11-08T08:20:37.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29200682f0 msgr2=0x7f292006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29200682f0 0x7f292006a7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2924000f60 tx=0x7f2924007dd0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 msgr2=0x7f293415fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 0x7f293415fb40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2934073d10 tx=0x7f292c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 shutdown_connections 2023-11-08T08:20:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29340b0880 0x7f293415fb40 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f29200682f0 0x7f292006a7a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f293406b7b0 0x7f293415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 --2- 172.21.15.116:0/2077159860 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f29340ad1f0 0x7f293415f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 >> 172.21.15.116:0/2077159860 conn(0x7f29340b2fc0 msgr2=0x7f29340aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 shutdown_connections 2023-11-08T08:20:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.437+0000 7f293940b700 1 -- 172.21.15.116:0/2077159860 wait complete. 2023-11-08T08:20:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:20:37.450 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2023-11-08T08:20:36.377308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":73348,"kb_used_data":5252,"kb_used_omap":0,"kb_used_meta":68032,"kb_avail":754901372,"statfs":{"total":773094113280,"available":773019004928,"internally_reserved":0,"allocated":5378048,"data_stored":2943639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69664768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.755001"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089068+0000","last_change":"2023-11-08T08:20:33.040952+0000","last_active":"2023-11-08T08:20:34.089068+0000","last_peered":"2023-11-08T08:20:34.089068+0000","last_clean":"2023-11-08T08:20:34.089068+0000","last_became_active":"2023-11-08T08:20:33.040646+0000","last_became_peered":"2023-11-08T08:20:33.040646+0000","last_unstale":"2023-11-08T08:20:34.089068+0000","last_undegraded":"2023-11-08T08:20:34.089068+0000","last_fullsized":"2023-11-08T08:20:34.089068+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087995+0000","last_change":"2023-11-08T08:20:33.026201+0000","last_active":"2023-11-08T08:20:34.087995+0000","last_peered":"2023-11-08T08:20:34.087995+0000","last_clean":"2023-11-08T08:20:34.087995+0000","last_became_active":"2023-11-08T08:20:33.025933+0000","last_became_peered":"2023-11-08T08:20:33.025933+0000","last_unstale":"2023-11-08T08:20:34.087995+0000","last_undegraded":"2023-11-08T08:20:34.087995+0000","last_fullsized":"2023-11-08T08:20:34.087995+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088583+0000","last_change":"2023-11-08T08:20:33.039111+0000","last_active":"2023-11-08T08:20:34.088583+0000","last_peered":"2023-11-08T08:20:34.088583+0000","last_clean":"2023-11-08T08:20:34.088583+0000","last_became_active":"2023-11-08T08:20:33.038823+0000","last_became_peered":"2023-11-08T08:20:33.038823+0000","last_unstale":"2023-11-08T08:20:34.088583+0000","last_undegraded":"2023-11-08T08:20:34.088583+0000","last_fullsized":"2023-11-08T08:20:34.088583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:52:48.726062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089092+0000","last_change":"2023-11-08T08:20:33.033970+0000","last_active":"2023-11-08T08:20:34.089092+0000","last_peered":"2023-11-08T08:20:34.089092+0000","last_clean":"2023-11-08T08:20:34.089092+0000","last_became_active":"2023-11-08T08:20:33.033693+0000","last_became_peered":"2023-11-08T08:20:33.033693+0000","last_unstale":"2023-11-08T08:20:34.089092+0000","last_undegraded":"2023-11-08T08:20:34.089092+0000","last_fullsized":"2023-11-08T08:20:34.089092+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:46:36.745520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087772+0000","last_change":"2023-11-08T08:20:33.031233+0000","last_active":"2023-11-08T08:20:34.087772+0000","last_peered":"2023-11-08T08:20:34.087772+0000","last_clean":"2023-11-08T08:20:34.087772+0000","last_became_active":"2023-11-08T08:20:33.030959+0000","last_became_peered":"2023-11-08T08:20:33.030959+0000","last_unstale":"2023-11-08T08:20:34.087772+0000","last_undegraded":"2023-11-08T08:20:34.087772+0000","last_fullsized":"2023-11-08T08:20:34.087772+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:56:33.178296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021405+0000","last_change":"2023-11-08T08:20:33.039791+0000","last_active":"2023-11-08T08:20:34.021405+0000","last_peered":"2023-11-08T08:20:34.021405+0000","last_clean":"2023-11-08T08:20:34.021405+0000","last_became_active":"2023-11-08T08:20:33.039509+0000","last_became_peered":"2023-11-08T08:20:33.039509+0000","last_unstale":"2023-11-08T08:20:34.021405+0000","last_undegraded":"2023-11-08T08:20:34.021405+0000","last_fullsized":"2023-11-08T08:20:34.021405+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021786+0000","last_change":"2023-11-08T08:20:33.040580+0000","last_active":"2023-11-08T08:20:34.021786+0000","last_peered":"2023-11-08T08:20:34.021786+0000","last_clean":"2023-11-08T08:20:34.021786+0000","last_became_active":"2023-11-08T08:20:33.040334+0000","last_became_peered":"2023-11-08T08:20:33.040334+0000","last_unstale":"2023-11-08T08:20:34.021786+0000","last_undegraded":"2023-11-08T08:20:34.021786+0000","last_fullsized":"2023-11-08T08:20:34.021786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021173+0000","last_change":"2023-11-08T08:20:33.047781+0000","last_active":"2023-11-08T08:20:34.021173+0000","last_peered":"2023-11-08T08:20:34.021173+0000","last_clean":"2023-11-08T08:20:34.021173+0000","last_became_active":"2023-11-08T08:20:33.047519+0000","last_became_peered":"2023-11-08T08:20:33.047519+0000","last_unstale":"2023-11-08T08:20:34.021173+0000","last_undegraded":"2023-11-08T08:20:34.021173+0000","last_fullsized":"2023-11-08T08:20:34.021173+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088850+0000","last_change":"2023-11-08T08:20:33.038697+0000","last_active":"2023-11-08T08:20:34.088850+0000","last_peered":"2023-11-08T08:20:34.088850+0000","last_clean":"2023-11-08T08:20:34.088850+0000","last_became_active":"2023-11-08T08:20:33.038400+0000","last_became_peered":"2023-11-08T08:20:33.038400+0000","last_unstale":"2023-11-08T08:20:34.088850+0000","last_undegraded":"2023-11-08T08:20:34.088850+0000","last_fullsized":"2023-11-08T08:20:34.088850+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:01:14.330615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022078+0000","last_change":"2023-11-08T08:20:33.034382+0000","last_active":"2023-11-08T08:20:34.022078+0000","last_peered":"2023-11-08T08:20:34.022078+0000","last_clean":"2023-11-08T08:20:34.022078+0000","last_became_active":"2023-11-08T08:20:33.034087+0000","last_became_peered":"2023-11-08T08:20:33.034087+0000","last_unstale":"2023-11-08T08:20:34.022078+0000","last_undegraded":"2023-11-08T08:20:34.022078+0000","last_fullsized":"2023-11-08T08:20:34.022078+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:20.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088798+0000","last_change":"2023-11-08T08:20:33.039836+0000","last_active":"2023-11-08T08:20:34.088798+0000","last_peered":"2023-11-08T08:20:34.088798+0000","last_clean":"2023-11-08T08:20:34.088798+0000","last_became_active":"2023-11-08T08:20:33.039507+0000","last_became_peered":"2023-11-08T08:20:33.039507+0000","last_unstale":"2023-11-08T08:20:34.088798+0000","last_undegraded":"2023-11-08T08:20:34.088798+0000","last_fullsized":"2023-11-08T08:20:34.088798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021849+0000","last_change":"2023-11-08T08:20:33.039388+0000","last_active":"2023-11-08T08:20:34.021849+0000","last_peered":"2023-11-08T08:20:34.021849+0000","last_clean":"2023-11-08T08:20:34.021849+0000","last_became_active":"2023-11-08T08:20:33.039070+0000","last_became_peered":"2023-11-08T08:20:33.039070+0000","last_unstale":"2023-11-08T08:20:34.021849+0000","last_undegraded":"2023-11-08T08:20:34.021849+0000","last_fullsized":"2023-11-08T08:20:34.021849+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:08:02.296246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021517+0000","last_change":"2023-11-08T08:20:33.041545+0000","last_active":"2023-11-08T08:20:34.021517+0000","last_peered":"2023-11-08T08:20:34.021517+0000","last_clean":"2023-11-08T08:20:34.021517+0000","last_became_active":"2023-11-08T08:20:33.041264+0000","last_became_peered":"2023-11-08T08:20:33.041264+0000","last_unstale":"2023-11-08T08:20:34.021517+0000","last_undegraded":"2023-11-08T08:20:34.021517+0000","last_fullsized":"2023-11-08T08:20:34.021517+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088473+0000","last_change":"2023-11-08T08:20:33.046425+0000","last_active":"2023-11-08T08:20:34.088473+0000","last_peered":"2023-11-08T08:20:34.088473+0000","last_clean":"2023-11-08T08:20:34.088473+0000","last_became_active":"2023-11-08T08:20:33.046123+0000","last_became_peered":"2023-11-08T08:20:33.046123+0000","last_unstale":"2023-11-08T08:20:34.088473+0000","last_undegraded":"2023-11-08T08:20:34.088473+0000","last_fullsized":"2023-11-08T08:20:34.088473+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088187+0000","last_change":"2023-11-08T08:20:33.036896+0000","last_active":"2023-11-08T08:20:34.088187+0000","last_peered":"2023-11-08T08:20:34.088187+0000","last_clean":"2023-11-08T08:20:34.088187+0000","last_became_active":"2023-11-08T08:20:33.036462+0000","last_became_peered":"2023-11-08T08:20:33.036462+0000","last_unstale":"2023-11-08T08:20:34.088187+0000","last_undegraded":"2023-11-08T08:20:34.088187+0000","last_fullsized":"2023-11-08T08:20:34.088187+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:11:16.998268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021173+0000","last_change":"2023-11-08T08:20:33.040515+0000","last_active":"2023-11-08T08:20:34.021173+0000","last_peered":"2023-11-08T08:20:34.021173+0000","last_clean":"2023-11-08T08:20:34.021173+0000","last_became_active":"2023-11-08T08:20:33.040221+0000","last_became_peered":"2023-11-08T08:20:33.040221+0000","last_unstale":"2023-11-08T08:20:34.021173+0000","last_undegraded":"2023-11-08T08:20:34.021173+0000","last_fullsized":"2023-11-08T08:20:34.021173+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089214+0000","last_change":"2023-11-08T08:20:33.037114+0000","last_active":"2023-11-08T08:20:34.089214+0000","last_peered":"2023-11-08T08:20:34.089214+0000","last_clean":"2023-11-08T08:20:34.089214+0000","last_became_active":"2023-11-08T08:20:33.036834+0000","last_became_peered":"2023-11-08T08:20:33.036834+0000","last_unstale":"2023-11-08T08:20:34.089214+0000","last_undegraded":"2023-11-08T08:20:34.089214+0000","last_fullsized":"2023-11-08T08:20:34.089214+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088528+0000","last_change":"2023-11-08T08:20:33.037687+0000","last_active":"2023-11-08T08:20:34.088528+0000","last_peered":"2023-11-08T08:20:34.088528+0000","last_clean":"2023-11-08T08:20:34.088528+0000","last_became_active":"2023-11-08T08:20:33.035179+0000","last_became_peered":"2023-11-08T08:20:33.035179+0000","last_unstale":"2023-11-08T08:20:34.088528+0000","last_undegraded":"2023-11-08T08:20:34.088528+0000","last_fullsized":"2023-11-08T08:20:34.088528+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:07:01.555604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088706+0000","last_change":"2023-11-08T08:20:33.034002+0000","last_active":"2023-11-08T08:20:34.088706+0000","last_peered":"2023-11-08T08:20:34.088706+0000","last_clean":"2023-11-08T08:20:34.088706+0000","last_became_active":"2023-11-08T08:20:33.033684+0000","last_became_peered":"2023-11-08T08:20:33.033684+0000","last_unstale":"2023-11-08T08:20:34.088706+0000","last_undegraded":"2023-11-08T08:20:34.088706+0000","last_fullsized":"2023-11-08T08:20:34.088706+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:45:41.548671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088009+0000","last_change":"2023-11-08T08:20:33.027046+0000","last_active":"2023-11-08T08:20:34.088009+0000","last_peered":"2023-11-08T08:20:34.088009+0000","last_clean":"2023-11-08T08:20:34.088009+0000","last_became_active":"2023-11-08T08:20:33.026791+0000","last_became_peered":"2023-11-08T08:20:33.026791+0000","last_unstale":"2023-11-08T08:20:34.088009+0000","last_undegraded":"2023-11-08T08:20:34.088009+0000","last_fullsized":"2023-11-08T08:20:34.088009+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088410+0000","last_change":"2023-11-08T08:20:33.037485+0000","last_active":"2023-11-08T08:20:34.088410+0000","last_peered":"2023-11-08T08:20:34.088410+0000","last_clean":"2023-11-08T08:20:34.088410+0000","last_became_active":"2023-11-08T08:20:33.035255+0000","last_became_peered":"2023-11-08T08:20:33.035255+0000","last_unstale":"2023-11-08T08:20:34.088410+0000","last_undegraded":"2023-11-08T08:20:34.088410+0000","last_fullsized":"2023-11-08T08:20:34.088410+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:10:47.744346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088785+0000","last_change":"2023-11-08T08:20:33.044479+0000","last_active":"2023-11-08T08:20:34.088785+0000","last_peered":"2023-11-08T08:20:34.088785+0000","last_clean":"2023-11-08T08:20:34.088785+0000","last_became_active":"2023-11-08T08:20:33.044208+0000","last_became_peered":"2023-11-08T08:20:33.044208+0000","last_unstale":"2023-11-08T08:20:34.088785+0000","last_undegraded":"2023-11-08T08:20:34.088785+0000","last_fullsized":"2023-11-08T08:20:34.088785+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:32:46.187671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088263+0000","last_change":"2023-11-08T08:20:33.034864+0000","last_active":"2023-11-08T08:20:34.088263+0000","last_peered":"2023-11-08T08:20:34.088263+0000","last_clean":"2023-11-08T08:20:34.088263+0000","last_became_active":"2023-11-08T08:20:33.033759+0000","last_became_peered":"2023-11-08T08:20:33.033759+0000","last_unstale":"2023-11-08T08:20:34.088263+0000","last_undegraded":"2023-11-08T08:20:34.088263+0000","last_fullsized":"2023-11-08T08:20:34.088263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:01:06.860105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088211+0000","last_change":"2023-11-08T08:20:33.045243+0000","last_active":"2023-11-08T08:20:34.088211+0000","last_peered":"2023-11-08T08:20:34.088211+0000","last_clean":"2023-11-08T08:20:34.088211+0000","last_became_active":"2023-11-08T08:20:33.044970+0000","last_became_peered":"2023-11-08T08:20:33.044970+0000","last_unstale":"2023-11-08T08:20:34.088211+0000","last_undegraded":"2023-11-08T08:20:34.088211+0000","last_fullsized":"2023-11-08T08:20:34.088211+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:45:11.193724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.089080+0000","last_change":"2023-11-08T08:20:33.046532+0000","last_active":"2023-11-08T08:20:34.089080+0000","last_peered":"2023-11-08T08:20:34.089080+0000","last_clean":"2023-11-08T08:20:34.089080+0000","last_became_active":"2023-11-08T08:20:33.046182+0000","last_became_peered":"2023-11-08T08:20:33.046182+0000","last_unstale":"2023-11-08T08:20:34.089080+0000","last_undegraded":"2023-11-08T08:20:34.089080+0000","last_fullsized":"2023-11-08T08:20:34.089080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088970+0000","last_change":"2023-11-08T08:20:33.044587+0000","last_active":"2023-11-08T08:20:34.088970+0000","last_peered":"2023-11-08T08:20:34.088970+0000","last_clean":"2023-11-08T08:20:34.088970+0000","last_became_active":"2023-11-08T08:20:33.044033+0000","last_became_peered":"2023-11-08T08:20:33.044033+0000","last_unstale":"2023-11-08T08:20:34.088970+0000","last_undegraded":"2023-11-08T08:20:34.088970+0000","last_fullsized":"2023-11-08T08:20:34.088970+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:43:40.912492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022278+0000","last_change":"2023-11-08T08:20:33.047351+0000","last_active":"2023-11-08T08:20:34.022278+0000","last_peered":"2023-11-08T08:20:34.022278+0000","last_clean":"2023-11-08T08:20:34.022278+0000","last_became_active":"2023-11-08T08:20:33.046969+0000","last_became_peered":"2023-11-08T08:20:33.046969+0000","last_unstale":"2023-11-08T08:20:34.022278+0000","last_undegraded":"2023-11-08T08:20:34.022278+0000","last_fullsized":"2023-11-08T08:20:34.022278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087934+0000","last_change":"2023-11-08T08:20:33.044167+0000","last_active":"2023-11-08T08:20:34.087934+0000","last_peered":"2023-11-08T08:20:34.087934+0000","last_clean":"2023-11-08T08:20:34.087934+0000","last_became_active":"2023-11-08T08:20:33.043978+0000","last_became_peered":"2023-11-08T08:20:33.043978+0000","last_unstale":"2023-11-08T08:20:34.087934+0000","last_undegraded":"2023-11-08T08:20:34.087934+0000","last_fullsized":"2023-11-08T08:20:34.087934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088438+0000","last_change":"2023-11-08T08:20:33.038706+0000","last_active":"2023-11-08T08:20:34.088438+0000","last_peered":"2023-11-08T08:20:34.088438+0000","last_clean":"2023-11-08T08:20:34.088438+0000","last_became_active":"2023-11-08T08:20:33.038438+0000","last_became_peered":"2023-11-08T08:20:33.038438+0000","last_unstale":"2023-11-08T08:20:34.088438+0000","last_undegraded":"2023-11-08T08:20:34.088438+0000","last_fullsized":"2023-11-08T08:20:34.088438+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021093+0000","last_change":"2023-11-08T08:20:33.037840+0000","last_active":"2023-11-08T08:20:34.021093+0000","last_peered":"2023-11-08T08:20:34.021093+0000","last_clean":"2023-11-08T08:20:34.021093+0000","last_became_active":"2023-11-08T08:20:33.037593+0000","last_became_peered":"2023-11-08T08:20:33.037593+0000","last_unstale":"2023-11-08T08:20:34.021093+0000","last_undegraded":"2023-11-08T08:20:34.021093+0000","last_fullsized":"2023-11-08T08:20:34.021093+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:00:34.853030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088368+0000","last_change":"2023-11-08T08:20:33.044048+0000","last_active":"2023-11-08T08:20:34.088368+0000","last_peered":"2023-11-08T08:20:34.088368+0000","last_clean":"2023-11-08T08:20:34.088368+0000","last_became_active":"2023-11-08T08:20:33.043735+0000","last_became_peered":"2023-11-08T08:20:33.043735+0000","last_unstale":"2023-11-08T08:20:34.088368+0000","last_undegraded":"2023-11-08T08:20:34.088368+0000","last_fullsized":"2023-11-08T08:20:34.088368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:36:37.828816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087559+0000","last_change":"2023-11-08T08:20:33.040511+0000","last_active":"2023-11-08T08:20:34.087559+0000","last_peered":"2023-11-08T08:20:34.087559+0000","last_clean":"2023-11-08T08:20:34.087559+0000","last_became_active":"2023-11-08T08:20:33.040226+0000","last_became_peered":"2023-11-08T08:20:33.040226+0000","last_unstale":"2023-11-08T08:20:34.087559+0000","last_undegraded":"2023-11-08T08:20:34.087559+0000","last_fullsized":"2023-11-08T08:20:34.087559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021092+0000","last_change":"2023-11-08T08:20:33.033215+0000","last_active":"2023-11-08T08:20:34.021092+0000","last_peered":"2023-11-08T08:20:34.021092+0000","last_clean":"2023-11-08T08:20:34.021092+0000","last_became_active":"2023-11-08T08:20:33.032935+0000","last_became_peered":"2023-11-08T08:20:33.032935+0000","last_unstale":"2023-11-08T08:20:34.021092+0000","last_undegraded":"2023-11-08T08:20:34.021092+0000","last_fullsized":"2023-11-08T08:20:34.021092+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:57:01.855036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"43'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088048+0000","last_change":"2023-11-08T08:20:32.647788+0000","last_active":"2023-11-08T08:20:34.088048+0000","last_peered":"2023-11-08T08:20:34.088048+0000","last_clean":"2023-11-08T08:20:34.088048+0000","last_became_active":"2023-11-08T08:20:32.647474+0000","last_became_peered":"2023-11-08T08:20:32.647474+0000","last_unstale":"2023-11-08T08:20:34.088048+0000","last_undegraded":"2023-11-08T08:20:34.088048+0000","last_fullsized":"2023-11-08T08:20:34.088048+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:36:36.511657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021064+0000","last_change":"2023-11-08T08:20:33.035667+0000","last_active":"2023-11-08T08:20:34.021064+0000","last_peered":"2023-11-08T08:20:34.021064+0000","last_clean":"2023-11-08T08:20:34.021064+0000","last_became_active":"2023-11-08T08:20:33.034101+0000","last_became_peered":"2023-11-08T08:20:33.034101+0000","last_unstale":"2023-11-08T08:20:34.021064+0000","last_undegraded":"2023-11-08T08:20:34.021064+0000","last_fullsized":"2023-11-08T08:20:34.021064+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:43:40.912492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022649+0000","last_change":"2023-11-08T08:20:32.649400+0000","last_active":"2023-11-08T08:20:34.022649+0000","last_peered":"2023-11-08T08:20:34.022649+0000","last_clean":"2023-11-08T08:20:34.022649+0000","last_became_active":"2023-11-08T08:20:32.649191+0000","last_became_peered":"2023-11-08T08:20:32.649191+0000","last_unstale":"2023-11-08T08:20:34.022649+0000","last_undegraded":"2023-11-08T08:20:34.022649+0000","last_fullsized":"2023-11-08T08:20:34.022649+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:55.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021207+0000","last_change":"2023-11-08T08:20:33.035274+0000","last_active":"2023-11-08T08:20:34.021207+0000","last_peered":"2023-11-08T08:20:34.021207+0000","last_clean":"2023-11-08T08:20:34.021207+0000","last_became_active":"2023-11-08T08:20:33.033839+0000","last_became_peered":"2023-11-08T08:20:33.033839+0000","last_unstale":"2023-11-08T08:20:34.021207+0000","last_undegraded":"2023-11-08T08:20:34.021207+0000","last_fullsized":"2023-11-08T08:20:34.021207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:44.931729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022141+0000","last_change":"2023-11-08T08:20:32.641284+0000","last_active":"2023-11-08T08:20:34.022141+0000","last_peered":"2023-11-08T08:20:34.022141+0000","last_clean":"2023-11-08T08:20:34.022141+0000","last_became_active":"2023-11-08T08:20:32.641014+0000","last_became_peered":"2023-11-08T08:20:32.641014+0000","last_unstale":"2023-11-08T08:20:34.022141+0000","last_undegraded":"2023-11-08T08:20:34.022141+0000","last_fullsized":"2023-11-08T08:20:34.022141+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:16.616131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021673+0000","last_change":"2023-11-08T08:20:33.036893+0000","last_active":"2023-11-08T08:20:34.021673+0000","last_peered":"2023-11-08T08:20:34.021673+0000","last_clean":"2023-11-08T08:20:34.021673+0000","last_became_active":"2023-11-08T08:20:33.036512+0000","last_became_peered":"2023-11-08T08:20:33.036512+0000","last_unstale":"2023-11-08T08:20:34.021673+0000","last_undegraded":"2023-11-08T08:20:34.021673+0000","last_fullsized":"2023-11-08T08:20:34.021673+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087904+0000","last_change":"2023-11-08T08:20:32.645234+0000","last_active":"2023-11-08T08:20:34.087904+0000","last_peered":"2023-11-08T08:20:34.087904+0000","last_clean":"2023-11-08T08:20:34.087904+0000","last_became_active":"2023-11-08T08:20:32.644949+0000","last_became_peered":"2023-11-08T08:20:32.644949+0000","last_unstale":"2023-11-08T08:20:34.087904+0000","last_undegraded":"2023-11-08T08:20:34.087904+0000","last_fullsized":"2023-11-08T08:20:34.087904+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:55.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087858+0000","last_change":"2023-11-08T08:20:33.025926+0000","last_active":"2023-11-08T08:20:34.087858+0000","last_peered":"2023-11-08T08:20:34.087858+0000","last_clean":"2023-11-08T08:20:34.087858+0000","last_became_active":"2023-11-08T08:20:33.025682+0000","last_became_peered":"2023-11-08T08:20:33.025682+0000","last_unstale":"2023-11-08T08:20:34.087858+0000","last_undegraded":"2023-11-08T08:20:34.087858+0000","last_fullsized":"2023-11-08T08:20:34.087858+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:56:33.178296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088310+0000","last_change":"2023-11-08T08:20:32.642448+0000","last_active":"2023-11-08T08:20:34.088310+0000","last_peered":"2023-11-08T08:20:34.088310+0000","last_clean":"2023-11-08T08:20:34.088310+0000","last_became_active":"2023-11-08T08:20:32.642135+0000","last_became_peered":"2023-11-08T08:20:32.642135+0000","last_unstale":"2023-11-08T08:20:34.088310+0000","last_undegraded":"2023-11-08T08:20:34.088310+0000","last_fullsized":"2023-11-08T08:20:34.088310+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:45.491146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021192+0000","last_change":"2023-11-08T08:20:33.035308+0000","last_active":"2023-11-08T08:20:34.021192+0000","last_peered":"2023-11-08T08:20:34.021192+0000","last_clean":"2023-11-08T08:20:34.021192+0000","last_became_active":"2023-11-08T08:20:33.035040+0000","last_became_peered":"2023-11-08T08:20:33.035040+0000","last_unstale":"2023-11-08T08:20:34.021192+0000","last_undegraded":"2023-11-08T08:20:34.021192+0000","last_fullsized":"2023-11-08T08:20:34.021192+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:13:11.268187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021304+0000","last_change":"2023-11-08T08:20:32.638835+0000","last_active":"2023-11-08T08:20:34.021304+0000","last_peered":"2023-11-08T08:20:34.021304+0000","last_clean":"2023-11-08T08:20:34.021304+0000","last_became_active":"2023-11-08T08:20:32.638553+0000","last_became_peered":"2023-11-08T08:20:32.638553+0000","last_unstale":"2023-11-08T08:20:34.021304+0000","last_undegraded":"2023-11-08T08:20:34.021304+0000","last_fullsized":"2023-11-08T08:20:34.021304+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:42:18.908130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022126+0000","last_change":"2023-11-08T08:20:33.037865+0000","last_active":"2023-11-08T08:20:34.022126+0000","last_peered":"2023-11-08T08:20:34.022126+0000","last_clean":"2023-11-08T08:20:34.022126+0000","last_became_active":"2023-11-08T08:20:33.037154+0000","last_became_peered":"2023-11-08T08:20:33.037154+0000","last_unstale":"2023-11-08T08:20:34.022126+0000","last_undegraded":"2023-11-08T08:20:34.022126+0000","last_fullsized":"2023-11-08T08:20:34.022126+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:49:25.775073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021051+0000","last_change":"2023-11-08T08:20:32.643968+0000","last_active":"2023-11-08T08:20:34.021051+0000","last_peered":"2023-11-08T08:20:34.021051+0000","last_clean":"2023-11-08T08:20:34.021051+0000","last_became_active":"2023-11-08T08:20:32.643610+0000","last_became_peered":"2023-11-08T08:20:32.643610+0000","last_unstale":"2023-11-08T08:20:34.021051+0000","last_undegraded":"2023-11-08T08:20:34.021051+0000","last_fullsized":"2023-11-08T08:20:34.021051+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:46:35.737983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088484+0000","last_change":"2023-11-08T08:20:33.042769+0000","last_active":"2023-11-08T08:20:34.088484+0000","last_peered":"2023-11-08T08:20:34.088484+0000","last_clean":"2023-11-08T08:20:34.088484+0000","last_became_active":"2023-11-08T08:20:33.042546+0000","last_became_peered":"2023-11-08T08:20:33.042546+0000","last_unstale":"2023-11-08T08:20:34.088484+0000","last_undegraded":"2023-11-08T08:20:34.088484+0000","last_fullsized":"2023-11-08T08:20:34.088484+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:42:19.915667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088077+0000","last_change":"2023-11-08T08:20:32.642541+0000","last_active":"2023-11-08T08:20:34.088077+0000","last_peered":"2023-11-08T08:20:34.088077+0000","last_clean":"2023-11-08T08:20:34.088077+0000","last_became_active":"2023-11-08T08:20:32.641890+0000","last_became_peered":"2023-11-08T08:20:32.641890+0000","last_unstale":"2023-11-08T08:20:34.088077+0000","last_undegraded":"2023-11-08T08:20:34.088077+0000","last_fullsized":"2023-11-08T08:20:34.088077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:21:08.720995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087402+0000","last_change":"2023-11-08T08:20:33.029212+0000","last_active":"2023-11-08T08:20:34.087402+0000","last_peered":"2023-11-08T08:20:34.087402+0000","last_clean":"2023-11-08T08:20:34.087402+0000","last_became_active":"2023-11-08T08:20:33.028931+0000","last_became_peered":"2023-11-08T08:20:33.028931+0000","last_unstale":"2023-11-08T08:20:34.087402+0000","last_undegraded":"2023-11-08T08:20:34.087402+0000","last_fullsized":"2023-11-08T08:20:34.087402+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088540+0000","last_change":"2023-11-08T08:20:32.638341+0000","last_active":"2023-11-08T08:20:34.088540+0000","last_peered":"2023-11-08T08:20:34.088540+0000","last_clean":"2023-11-08T08:20:34.088540+0000","last_became_active":"2023-11-08T08:20:32.638066+0000","last_became_peered":"2023-11-08T08:20:32.638066+0000","last_unstale":"2023-11-08T08:20:34.088540+0000","last_undegraded":"2023-11-08T08:20:34.088540+0000","last_fullsized":"2023-11-08T08:20:34.088540+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:38.025405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088361+0000","last_change":"2023-11-08T08:20:33.034928+0000","last_active":"2023-11-08T08:20:34.088361+0000","last_peered":"2023-11-08T08:20:34.088361+0000","last_clean":"2023-11-08T08:20:34.088361+0000","last_became_active":"2023-11-08T08:20:33.034671+0000","last_became_peered":"2023-11-08T08:20:33.034671+0000","last_unstale":"2023-11-08T08:20:34.088361+0000","last_undegraded":"2023-11-08T08:20:34.088361+0000","last_fullsized":"2023-11-08T08:20:34.088361+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:21:09.728532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088577+0000","last_change":"2023-11-08T08:20:32.645389+0000","last_active":"2023-11-08T08:20:34.088577+0000","last_peered":"2023-11-08T08:20:34.088577+0000","last_clean":"2023-11-08T08:20:34.088577+0000","last_became_active":"2023-11-08T08:20:32.645090+0000","last_became_peered":"2023-11-08T08:20:32.645090+0000","last_unstale":"2023-11-08T08:20:34.088577+0000","last_undegraded":"2023-11-08T08:20:34.088577+0000","last_fullsized":"2023-11-08T08:20:34.088577+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:19.636821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088582+0000","last_change":"2023-11-08T08:20:33.037707+0000","last_active":"2023-11-08T08:20:34.088582+0000","last_peered":"2023-11-08T08:20:34.088582+0000","last_clean":"2023-11-08T08:20:34.088582+0000","last_became_active":"2023-11-08T08:20:33.037444+0000","last_became_peered":"2023-11-08T08:20:33.037444+0000","last_unstale":"2023-11-08T08:20:34.088582+0000","last_undegraded":"2023-11-08T08:20:34.088582+0000","last_fullsized":"2023-11-08T08:20:34.088582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"43'3","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021292+0000","last_change":"2023-11-08T08:20:32.641224+0000","last_active":"2023-11-08T08:20:34.021292+0000","last_peered":"2023-11-08T08:20:34.021292+0000","last_clean":"2023-11-08T08:20:34.021292+0000","last_became_active":"2023-11-08T08:20:32.640873+0000","last_became_peered":"2023-11-08T08:20:32.640873+0000","last_unstale":"2023-11-08T08:20:34.021292+0000","last_undegraded":"2023-11-08T08:20:34.021292+0000","last_fullsized":"2023-11-08T08:20:34.021292+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:07:00.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087887+0000","last_change":"2023-11-08T08:20:33.026829+0000","last_active":"2023-11-08T08:20:34.087887+0000","last_peered":"2023-11-08T08:20:34.087887+0000","last_clean":"2023-11-08T08:20:34.087887+0000","last_became_active":"2023-11-08T08:20:33.026569+0000","last_became_peered":"2023-11-08T08:20:33.026569+0000","last_unstale":"2023-11-08T08:20:34.087887+0000","last_undegraded":"2023-11-08T08:20:34.087887+0000","last_fullsized":"2023-11-08T08:20:34.087887+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:42:30.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"43'6","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022169+0000","last_change":"2023-11-08T08:20:32.646660+0000","last_active":"2023-11-08T08:20:34.022169+0000","last_peered":"2023-11-08T08:20:34.022169+0000","last_clean":"2023-11-08T08:20:34.022169+0000","last_became_active":"2023-11-08T08:20:32.645049+0000","last_became_peered":"2023-11-08T08:20:32.645049+0000","last_unstale":"2023-11-08T08:20:34.022169+0000","last_undegraded":"2023-11-08T08:20:34.022169+0000","last_fullsized":"2023-11-08T08:20:34.022169+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:14:18.361427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021338+0000","last_change":"2023-11-08T08:20:33.035535+0000","last_active":"2023-11-08T08:20:34.021338+0000","last_peered":"2023-11-08T08:20:34.021338+0000","last_clean":"2023-11-08T08:20:34.021338+0000","last_became_active":"2023-11-08T08:20:33.034052+0000","last_became_peered":"2023-11-08T08:20:33.034052+0000","last_unstale":"2023-11-08T08:20:34.021338+0000","last_undegraded":"2023-11-08T08:20:34.021338+0000","last_fullsized":"2023-11-08T08:20:34.021338+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:04:10.931634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021716+0000","last_change":"2023-11-08T08:20:32.646559+0000","last_active":"2023-11-08T08:20:34.021716+0000","last_peered":"2023-11-08T08:20:34.021716+0000","last_clean":"2023-11-08T08:20:34.021716+0000","last_became_active":"2023-11-08T08:20:32.646258+0000","last_became_peered":"2023-11-08T08:20:32.646258+0000","last_unstale":"2023-11-08T08:20:34.021716+0000","last_undegraded":"2023-11-08T08:20:34.021716+0000","last_fullsized":"2023-11-08T08:20:34.021716+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:43:45.224233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088899+0000","last_change":"2023-11-08T08:20:33.046636+0000","last_active":"2023-11-08T08:20:34.088899+0000","last_peered":"2023-11-08T08:20:34.088899+0000","last_clean":"2023-11-08T08:20:34.088899+0000","last_became_active":"2023-11-08T08:20:33.046400+0000","last_became_peered":"2023-11-08T08:20:33.046400+0000","last_unstale":"2023-11-08T08:20:34.088899+0000","last_undegraded":"2023-11-08T08:20:34.088899+0000","last_fullsized":"2023-11-08T08:20:34.088899+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:51:02.264321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088386+0000","last_change":"2023-11-08T08:20:32.642090+0000","last_active":"2023-11-08T08:20:34.088386+0000","last_peered":"2023-11-08T08:20:34.088386+0000","last_clean":"2023-11-08T08:20:34.088386+0000","last_became_active":"2023-11-08T08:20:32.641772+0000","last_became_peered":"2023-11-08T08:20:32.641772+0000","last_unstale":"2023-11-08T08:20:34.088386+0000","last_undegraded":"2023-11-08T08:20:34.088386+0000","last_fullsized":"2023-11-08T08:20:34.088386+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:22.535882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022530+0000","last_change":"2023-11-08T08:20:33.039967+0000","last_active":"2023-11-08T08:20:34.022530+0000","last_peered":"2023-11-08T08:20:34.022530+0000","last_clean":"2023-11-08T08:20:34.022530+0000","last_became_active":"2023-11-08T08:20:33.039641+0000","last_became_peered":"2023-11-08T08:20:33.039641+0000","last_unstale":"2023-11-08T08:20:34.022530+0000","last_undegraded":"2023-11-08T08:20:34.022530+0000","last_fullsized":"2023-11-08T08:20:34.022530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:37:44.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087779+0000","last_change":"2023-11-08T08:20:32.641027+0000","last_active":"2023-11-08T08:20:34.087779+0000","last_peered":"2023-11-08T08:20:34.087779+0000","last_clean":"2023-11-08T08:20:34.087779+0000","last_became_active":"2023-11-08T08:20:32.640691+0000","last_became_peered":"2023-11-08T08:20:32.640691+0000","last_unstale":"2023-11-08T08:20:34.087779+0000","last_undegraded":"2023-11-08T08:20:34.087779+0000","last_fullsized":"2023-11-08T08:20:34.087779+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:43:45.224233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088908+0000","last_change":"2023-11-08T08:20:33.034299+0000","last_active":"2023-11-08T08:20:34.088908+0000","last_peered":"2023-11-08T08:20:34.088908+0000","last_clean":"2023-11-08T08:20:34.088908+0000","last_became_active":"2023-11-08T08:20:33.033821+0000","last_became_peered":"2023-11-08T08:20:33.033821+0000","last_unstale":"2023-11-08T08:20:34.088908+0000","last_undegraded":"2023-11-08T08:20:34.088908+0000","last_fullsized":"2023-11-08T08:20:34.088908+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:43:15.255479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"43'8","reported_seq":19,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021270+0000","last_change":"2023-11-08T08:20:32.647360+0000","last_active":"2023-11-08T08:20:34.021270+0000","last_peered":"2023-11-08T08:20:34.021270+0000","last_clean":"2023-11-08T08:20:34.021270+0000","last_became_active":"2023-11-08T08:20:32.647033+0000","last_became_peered":"2023-11-08T08:20:32.647033+0000","last_unstale":"2023-11-08T08:20:34.021270+0000","last_undegraded":"2023-11-08T08:20:34.021270+0000","last_fullsized":"2023-11-08T08:20:34.021270+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:55:30.574115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021582+0000","last_change":"2023-11-08T08:20:33.029834+0000","last_active":"2023-11-08T08:20:34.021582+0000","last_peered":"2023-11-08T08:20:34.021582+0000","last_clean":"2023-11-08T08:20:34.021582+0000","last_became_active":"2023-11-08T08:20:33.029598+0000","last_became_peered":"2023-11-08T08:20:33.029598+0000","last_unstale":"2023-11-08T08:20:34.021582+0000","last_undegraded":"2023-11-08T08:20:34.021582+0000","last_fullsized":"2023-11-08T08:20:34.021582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:59:39.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022722+0000","last_change":"2023-11-08T08:20:32.649522+0000","last_active":"2023-11-08T08:20:34.022722+0000","last_peered":"2023-11-08T08:20:34.022722+0000","last_clean":"2023-11-08T08:20:34.022722+0000","last_became_active":"2023-11-08T08:20:32.649239+0000","last_became_peered":"2023-11-08T08:20:32.649239+0000","last_unstale":"2023-11-08T08:20:34.022722+0000","last_undegraded":"2023-11-08T08:20:34.022722+0000","last_fullsized":"2023-11-08T08:20:34.022722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:14:18.361427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088106+0000","last_change":"2023-11-08T08:20:33.038087+0000","last_active":"2023-11-08T08:20:34.088106+0000","last_peered":"2023-11-08T08:20:34.088106+0000","last_clean":"2023-11-08T08:20:34.088106+0000","last_became_active":"2023-11-08T08:20:33.037806+0000","last_became_peered":"2023-11-08T08:20:33.037806+0000","last_unstale":"2023-11-08T08:20:34.088106+0000","last_undegraded":"2023-11-08T08:20:34.088106+0000","last_fullsized":"2023-11-08T08:20:34.088106+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:23.543419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022638+0000","last_change":"2023-11-08T08:20:32.649879+0000","last_active":"2023-11-08T08:20:34.022638+0000","last_peered":"2023-11-08T08:20:34.022638+0000","last_clean":"2023-11-08T08:20:34.022638+0000","last_became_active":"2023-11-08T08:20:32.649619+0000","last_became_peered":"2023-11-08T08:20:32.649619+0000","last_unstale":"2023-11-08T08:20:34.022638+0000","last_undegraded":"2023-11-08T08:20:34.022638+0000","last_fullsized":"2023-11-08T08:20:34.022638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:47:24.700788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087562+0000","last_change":"2023-11-08T08:20:33.027549+0000","last_active":"2023-11-08T08:20:34.087562+0000","last_peered":"2023-11-08T08:20:34.087562+0000","last_clean":"2023-11-08T08:20:34.087562+0000","last_became_active":"2023-11-08T08:20:33.027312+0000","last_became_peered":"2023-11-08T08:20:33.027312+0000","last_unstale":"2023-11-08T08:20:34.087562+0000","last_undegraded":"2023-11-08T08:20:34.087562+0000","last_fullsized":"2023-11-08T08:20:34.087562+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"43'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088652+0000","last_change":"2023-11-08T08:20:32.642896+0000","last_active":"2023-11-08T08:20:34.088652+0000","last_peered":"2023-11-08T08:20:34.088652+0000","last_clean":"2023-11-08T08:20:34.088652+0000","last_became_active":"2023-11-08T08:20:32.642674+0000","last_became_peered":"2023-11-08T08:20:32.642674+0000","last_unstale":"2023-11-08T08:20:34.088652+0000","last_undegraded":"2023-11-08T08:20:34.088652+0000","last_fullsized":"2023-11-08T08:20:34.088652+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:21.875398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021609+0000","last_change":"2023-11-08T08:20:33.047525+0000","last_active":"2023-11-08T08:20:34.021609+0000","last_peered":"2023-11-08T08:20:34.021609+0000","last_clean":"2023-11-08T08:20:34.021609+0000","last_became_active":"2023-11-08T08:20:33.047223+0000","last_became_peered":"2023-11-08T08:20:33.047223+0000","last_unstale":"2023-11-08T08:20:34.021609+0000","last_undegraded":"2023-11-08T08:20:34.021609+0000","last_fullsized":"2023-11-08T08:20:34.021609+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:42:30.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087763+0000","last_change":"2023-11-08T08:20:32.641498+0000","last_active":"2023-11-08T08:20:34.087763+0000","last_peered":"2023-11-08T08:20:34.087763+0000","last_clean":"2023-11-08T08:20:34.087763+0000","last_became_active":"2023-11-08T08:20:32.641273+0000","last_became_peered":"2023-11-08T08:20:32.641273+0000","last_unstale":"2023-11-08T08:20:34.087763+0000","last_undegraded":"2023-11-08T08:20:34.087763+0000","last_fullsized":"2023-11-08T08:20:34.087763+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:55:30.574115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087865+0000","last_change":"2023-11-08T08:20:33.031258+0000","last_active":"2023-11-08T08:20:34.087865+0000","last_peered":"2023-11-08T08:20:34.087865+0000","last_clean":"2023-11-08T08:20:34.087865+0000","last_became_active":"2023-11-08T08:20:33.031004+0000","last_became_peered":"2023-11-08T08:20:33.031004+0000","last_unstale":"2023-11-08T08:20:34.087865+0000","last_undegraded":"2023-11-08T08:20:34.087865+0000","last_fullsized":"2023-11-08T08:20:34.087865+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:19:07.596651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088409+0000","last_change":"2023-11-08T08:20:32.645648+0000","last_active":"2023-11-08T08:20:34.088409+0000","last_peered":"2023-11-08T08:20:34.088409+0000","last_clean":"2023-11-08T08:20:34.088409+0000","last_became_active":"2023-11-08T08:20:32.645357+0000","last_became_peered":"2023-11-08T08:20:32.645357+0000","last_unstale":"2023-11-08T08:20:34.088409+0000","last_undegraded":"2023-11-08T08:20:34.088409+0000","last_fullsized":"2023-11-08T08:20:34.088409+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:16.616131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088566+0000","last_change":"2023-11-08T08:20:33.035942+0000","last_active":"2023-11-08T08:20:34.088566+0000","last_peered":"2023-11-08T08:20:34.088566+0000","last_clean":"2023-11-08T08:20:34.088566+0000","last_became_active":"2023-11-08T08:20:33.035195+0000","last_became_peered":"2023-11-08T08:20:33.035195+0000","last_unstale":"2023-11-08T08:20:34.088566+0000","last_undegraded":"2023-11-08T08:20:34.088566+0000","last_fullsized":"2023-11-08T08:20:34.088566+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:11:41.599485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088789+0000","last_change":"2023-11-08T08:20:32.644955+0000","last_active":"2023-11-08T08:20:34.088789+0000","last_peered":"2023-11-08T08:20:34.088789+0000","last_clean":"2023-11-08T08:20:34.088789+0000","last_became_active":"2023-11-08T08:20:32.644690+0000","last_became_peered":"2023-11-08T08:20:32.644690+0000","last_unstale":"2023-11-08T08:20:34.088789+0000","last_undegraded":"2023-11-08T08:20:34.088789+0000","last_fullsized":"2023-11-08T08:20:34.088789+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:42:18.908130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088379+0000","last_change":"2023-11-08T08:20:33.037993+0000","last_active":"2023-11-08T08:20:34.088379+0000","last_peered":"2023-11-08T08:20:34.088379+0000","last_clean":"2023-11-08T08:20:34.088379+0000","last_became_active":"2023-11-08T08:20:33.037173+0000","last_became_peered":"2023-11-08T08:20:33.037173+0000","last_unstale":"2023-11-08T08:20:34.088379+0000","last_undegraded":"2023-11-08T08:20:34.088379+0000","last_fullsized":"2023-11-08T08:20:34.088379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:08:02.296246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021822+0000","last_change":"2023-11-08T08:20:32.644960+0000","last_active":"2023-11-08T08:20:34.021822+0000","last_peered":"2023-11-08T08:20:34.021822+0000","last_clean":"2023-11-08T08:20:34.021822+0000","last_became_active":"2023-11-08T08:20:32.642226+0000","last_became_peered":"2023-11-08T08:20:32.642226+0000","last_unstale":"2023-11-08T08:20:34.021822+0000","last_undegraded":"2023-11-08T08:20:34.021822+0000","last_fullsized":"2023-11-08T08:20:34.021822+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:48:55.693665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022664+0000","last_change":"2023-11-08T08:20:33.044543+0000","last_active":"2023-11-08T08:20:34.022664+0000","last_peered":"2023-11-08T08:20:34.022664+0000","last_clean":"2023-11-08T08:20:34.022664+0000","last_became_active":"2023-11-08T08:20:33.044282+0000","last_became_peered":"2023-11-08T08:20:33.044282+0000","last_unstale":"2023-11-08T08:20:34.022664+0000","last_undegraded":"2023-11-08T08:20:34.022664+0000","last_fullsized":"2023-11-08T08:20:34.022664+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:13:23.543419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021581+0000","last_change":"2023-11-08T08:20:32.644350+0000","last_active":"2023-11-08T08:20:34.021581+0000","last_peered":"2023-11-08T08:20:34.021581+0000","last_clean":"2023-11-08T08:20:34.021581+0000","last_became_active":"2023-11-08T08:20:32.641483+0000","last_became_peered":"2023-11-08T08:20:32.641483+0000","last_unstale":"2023-11-08T08:20:34.021581+0000","last_undegraded":"2023-11-08T08:20:34.021581+0000","last_fullsized":"2023-11-08T08:20:34.021581+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:31:18.529121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022029+0000","last_change":"2023-11-08T08:20:33.039195+0000","last_active":"2023-11-08T08:20:34.022029+0000","last_peered":"2023-11-08T08:20:34.022029+0000","last_clean":"2023-11-08T08:20:34.022029+0000","last_became_active":"2023-11-08T08:20:33.038954+0000","last_became_peered":"2023-11-08T08:20:33.038954+0000","last_unstale":"2023-11-08T08:20:34.022029+0000","last_undegraded":"2023-11-08T08:20:34.022029+0000","last_fullsized":"2023-11-08T08:20:34.022029+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021729+0000","last_change":"2023-11-08T08:20:32.640819+0000","last_active":"2023-11-08T08:20:34.021729+0000","last_peered":"2023-11-08T08:20:34.021729+0000","last_clean":"2023-11-08T08:20:34.021729+0000","last_became_active":"2023-11-08T08:20:32.640305+0000","last_became_peered":"2023-11-08T08:20:32.640305+0000","last_unstale":"2023-11-08T08:20:34.021729+0000","last_undegraded":"2023-11-08T08:20:34.021729+0000","last_fullsized":"2023-11-08T08:20:34.021729+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:55:30.574115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.016572+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:20:33.016572+0000","last_peered":"2023-11-08T08:20:33.016572+0000","last_clean":"2023-11-08T08:20:33.016572+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:20:33.016572+0000","last_undegraded":"2023-11-08T08:20:33.016572+0000","last_fullsized":"2023-11-08T08:20:33.016572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088228+0000","last_change":"2023-11-08T08:20:33.027420+0000","last_active":"2023-11-08T08:20:34.088228+0000","last_peered":"2023-11-08T08:20:34.088228+0000","last_clean":"2023-11-08T08:20:34.088228+0000","last_became_active":"2023-11-08T08:20:33.027182+0000","last_became_peered":"2023-11-08T08:20:33.027182+0000","last_unstale":"2023-11-08T08:20:34.088228+0000","last_undegraded":"2023-11-08T08:20:34.088228+0000","last_fullsized":"2023-11-08T08:20:34.088228+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:07:58.396470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088902+0000","last_change":"2023-11-08T08:20:32.645392+0000","last_active":"2023-11-08T08:20:34.088902+0000","last_peered":"2023-11-08T08:20:34.088902+0000","last_clean":"2023-11-08T08:20:34.088902+0000","last_became_active":"2023-11-08T08:20:32.644516+0000","last_became_peered":"2023-11-08T08:20:32.644516+0000","last_unstale":"2023-11-08T08:20:34.088902+0000","last_undegraded":"2023-11-08T08:20:34.088902+0000","last_fullsized":"2023-11-08T08:20:34.088902+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:07:00.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.029305+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:20:33.029305+0000","last_peered":"2023-11-08T08:20:33.029305+0000","last_clean":"2023-11-08T08:20:33.029305+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:20:33.029305+0000","last_undegraded":"2023-11-08T08:20:33.029305+0000","last_fullsized":"2023-11-08T08:20:33.029305+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022217+0000","last_change":"2023-11-08T08:20:33.036789+0000","last_active":"2023-11-08T08:20:34.022217+0000","last_peered":"2023-11-08T08:20:34.022217+0000","last_clean":"2023-11-08T08:20:34.022217+0000","last_became_active":"2023-11-08T08:20:33.036498+0000","last_became_peered":"2023-11-08T08:20:33.036498+0000","last_unstale":"2023-11-08T08:20:34.022217+0000","last_undegraded":"2023-11-08T08:20:34.022217+0000","last_fullsized":"2023-11-08T08:20:34.022217+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:08:02.296246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021335+0000","last_change":"2023-11-08T08:20:32.641457+0000","last_active":"2023-11-08T08:20:34.021335+0000","last_peered":"2023-11-08T08:20:34.021335+0000","last_clean":"2023-11-08T08:20:34.021335+0000","last_became_active":"2023-11-08T08:20:32.641214+0000","last_became_peered":"2023-11-08T08:20:32.641214+0000","last_unstale":"2023-11-08T08:20:34.021335+0000","last_undegraded":"2023-11-08T08:20:34.021335+0000","last_fullsized":"2023-11-08T08:20:34.021335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:45.491146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.025713+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:20:33.025713+0000","last_peered":"2023-11-08T08:20:33.025713+0000","last_clean":"2023-11-08T08:20:33.025713+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:20:33.025713+0000","last_undegraded":"2023-11-08T08:20:33.025713+0000","last_fullsized":"2023-11-08T08:20:33.025713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":187,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.023225+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:20:33.023225+0000","last_peered":"2023-11-08T08:20:33.023225+0000","last_clean":"2023-11-08T08:20:33.023225+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:20:33.023225+0000","last_undegraded":"2023-11-08T08:20:33.023225+0000","last_fullsized":"2023-11-08T08:20:33.023225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088423+0000","last_change":"2023-11-08T08:20:33.035325+0000","last_active":"2023-11-08T08:20:34.088423+0000","last_peered":"2023-11-08T08:20:34.088423+0000","last_clean":"2023-11-08T08:20:34.088423+0000","last_became_active":"2023-11-08T08:20:33.035014+0000","last_became_peered":"2023-11-08T08:20:33.035014+0000","last_unstale":"2023-11-08T08:20:34.088423+0000","last_undegraded":"2023-11-08T08:20:34.088423+0000","last_fullsized":"2023-11-08T08:20:34.088423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:00:34.853030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088139+0000","last_change":"2023-11-08T08:20:32.644969+0000","last_active":"2023-11-08T08:20:34.088139+0000","last_peered":"2023-11-08T08:20:34.088139+0000","last_clean":"2023-11-08T08:20:34.088139+0000","last_became_active":"2023-11-08T08:20:32.644727+0000","last_became_peered":"2023-11-08T08:20:32.644727+0000","last_unstale":"2023-11-08T08:20:34.088139+0000","last_undegraded":"2023-11-08T08:20:34.088139+0000","last_fullsized":"2023-11-08T08:20:34.088139+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:46:35.737983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.020635+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:20:33.020635+0000","last_peered":"2023-11-08T08:20:33.020635+0000","last_clean":"2023-11-08T08:20:33.020635+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:20:33.020635+0000","last_undegraded":"2023-11-08T08:20:33.020635+0000","last_fullsized":"2023-11-08T08:20:33.020635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087469+0000","last_change":"2023-11-08T08:20:33.026312+0000","last_active":"2023-11-08T08:20:34.087469+0000","last_peered":"2023-11-08T08:20:34.087469+0000","last_clean":"2023-11-08T08:20:34.087469+0000","last_became_active":"2023-11-08T08:20:33.026055+0000","last_became_peered":"2023-11-08T08:20:33.026055+0000","last_unstale":"2023-11-08T08:20:34.087469+0000","last_undegraded":"2023-11-08T08:20:34.087469+0000","last_fullsized":"2023-11-08T08:20:34.087469+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:08:14.386212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022262+0000","last_change":"2023-11-08T08:20:32.640456+0000","last_active":"2023-11-08T08:20:34.022262+0000","last_peered":"2023-11-08T08:20:34.022262+0000","last_clean":"2023-11-08T08:20:34.022262+0000","last_became_active":"2023-11-08T08:20:32.640147+0000","last_became_peered":"2023-11-08T08:20:32.640147+0000","last_unstale":"2023-11-08T08:20:34.022262+0000","last_undegraded":"2023-11-08T08:20:34.022262+0000","last_fullsized":"2023-11-08T08:20:34.022262+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:27:43.728859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.028736+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:20:33.028736+0000","last_peered":"2023-11-08T08:20:33.028736+0000","last_clean":"2023-11-08T08:20:33.028736+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:20:33.028736+0000","last_undegraded":"2023-11-08T08:20:33.028736+0000","last_fullsized":"2023-11-08T08:20:33.028736+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088252+0000","last_change":"2023-11-08T08:20:33.046426+0000","last_active":"2023-11-08T08:20:34.088252+0000","last_peered":"2023-11-08T08:20:34.088252+0000","last_clean":"2023-11-08T08:20:34.088252+0000","last_became_active":"2023-11-08T08:20:33.046206+0000","last_became_peered":"2023-11-08T08:20:33.046206+0000","last_unstale":"2023-11-08T08:20:34.088252+0000","last_undegraded":"2023-11-08T08:20:34.088252+0000","last_fullsized":"2023-11-08T08:20:34.088252+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:46.498683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"43'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.021181+0000","last_change":"2023-11-08T08:20:32.644433+0000","last_active":"2023-11-08T08:20:34.021181+0000","last_peered":"2023-11-08T08:20:34.021181+0000","last_clean":"2023-11-08T08:20:34.021181+0000","last_became_active":"2023-11-08T08:20:32.644229+0000","last_became_peered":"2023-11-08T08:20:32.644229+0000","last_unstale":"2023-11-08T08:20:34.021181+0000","last_undegraded":"2023-11-08T08:20:34.021181+0000","last_fullsized":"2023-11-08T08:20:34.021181+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:21:08.720995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.023324+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:20:33.023324+0000","last_peered":"2023-11-08T08:20:33.023324+0000","last_clean":"2023-11-08T08:20:33.023324+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:20:33.023324+0000","last_undegraded":"2023-11-08T08:20:33.023324+0000","last_fullsized":"2023-11-08T08:20:33.023324+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022645+0000","last_change":"2023-11-08T08:20:33.042016+0000","last_active":"2023-11-08T08:20:34.022645+0000","last_peered":"2023-11-08T08:20:34.022645+0000","last_clean":"2023-11-08T08:20:34.022645+0000","last_became_active":"2023-11-08T08:20:33.040745+0000","last_became_peered":"2023-11-08T08:20:33.040745+0000","last_unstale":"2023-11-08T08:20:34.022645+0000","last_undegraded":"2023-11-08T08:20:34.022645+0000","last_fullsized":"2023-11-08T08:20:34.022645+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:22.882935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.088093+0000","last_change":"2023-11-08T08:20:32.642179+0000","last_active":"2023-11-08T08:20:34.088093+0000","last_peered":"2023-11-08T08:20:34.088093+0000","last_clean":"2023-11-08T08:20:34.088093+0000","last_became_active":"2023-11-08T08:20:32.641860+0000","last_became_peered":"2023-11-08T08:20:32.641860+0000","last_unstale":"2023-11-08T08:20:34.088093+0000","last_undegraded":"2023-11-08T08:20:34.088093+0000","last_fullsized":"2023-11-08T08:20:34.088093+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:51:01.256784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.023048+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:20:33.023048+0000","last_peered":"2023-11-08T08:20:33.023048+0000","last_clean":"2023-11-08T08:20:33.023048+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:20:33.023048+0000","last_undegraded":"2023-11-08T08:20:33.023048+0000","last_fullsized":"2023-11-08T08:20:33.023048+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.022265+0000","last_change":"2023-11-08T08:20:33.036733+0000","last_active":"2023-11-08T08:20:34.022265+0000","last_peered":"2023-11-08T08:20:34.022265+0000","last_clean":"2023-11-08T08:20:34.022265+0000","last_became_active":"2023-11-08T08:20:33.036448+0000","last_became_peered":"2023-11-08T08:20:33.036448+0000","last_unstale":"2023-11-08T08:20:34.022265+0000","last_undegraded":"2023-11-08T08:20:34.022265+0000","last_fullsized":"2023-11-08T08:20:34.022265+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:32.620016+0000","last_clean_scrub_stamp":"2023-11-08T08:20:32.620016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:42:30.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-11-08T08:20:34.087865+0000","last_change":"2023-11-08T08:20:32.637682+0000","last_active":"2023-11-08T08:20:34.087865+0000","last_peered":"2023-11-08T08:20:34.087865+0000","last_clean":"2023-11-08T08:20:34.087865+0000","last_became_active":"2023-11-08T08:20:32.637425+0000","last_became_peered":"2023-11-08T08:20:32.637425+0000","last_unstale":"2023-11-08T08:20:34.087865+0000","last_undegraded":"2023-11-08T08:20:34.087865+0000","last_fullsized":"2023-11-08T08:20:34.087865+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:20:31.612479+0000","last_clean_scrub_stamp":"2023-11-08T08:20:31.612479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:14:18.361427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-11-08T08:20:33.030141+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:20:33.030141+0000","last_peered":"2023-11-08T08:20:33.030141+0000","last_clean":"2023-11-08T08:20:33.030141+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:20:33.030141+0000","last_undegraded":"2023-11-08T08:20:33.030141+0000","last_fullsized":"2023-11-08T08:20:33.030141+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8776,"kb_used_data":512,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363064,"statfs":{"total":96636764160,"available":96627777536,"internally_reserved":0,"allocated":524288,"data_stored":219994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.45600000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":5.8410000000000002,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.54100000000000004}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":5.9059999999999997,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":5.8869999999999996,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.624}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.69799999999999995}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66600000000000004}]},{"osd":6,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.65100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8832,"kb_used_data":504,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363008,"statfs":{"total":96636764160,"available":96627720192,"internally_reserved":0,"allocated":516096,"data_stored":219243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45800000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.433}]},{"osd":3,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.51200000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.5},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.58599999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8900,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362940,"statfs":{"total":96636764160,"available":96627650560,"internally_reserved":0,"allocated":520192,"data_stored":220008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.252},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.309}]},{"osd":4,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.371}]},{"osd":5,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.33100000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42499999999999999}]},{"osd":7,"last update":"Wed Nov 8 08:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.36699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8836,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363004,"statfs":{"total":96636764160,"available":96627716096,"internally_reserved":0,"allocated":520192,"data_stored":220504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.434},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.47399999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.72299999999999998}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.628}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.65200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8900,"kb_used_data":508,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362940,"statfs":{"total":96636764160,"available":96627650560,"internally_reserved":0,"allocated":520192,"data_stored":219972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.314}]},{"osd":3,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"last":0.379}]},{"osd":4,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54200000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.621}]},{"osd":6,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66900000000000004}]},{"osd":7,"last update":"Wed Nov 8 08:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10056,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94361784,"statfs":{"total":96636764160,"available":96626466816,"internally_reserved":0,"allocated":1114112,"data_stored":810396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.5070000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.502},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.56599999999999995}]},{"osd":4,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.876},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.76000000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.91500000000000004}]},{"osd":7,"last update":"Wed Nov 8 08:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.85399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9108,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362732,"statfs":{"total":96636764160,"available":96627437568,"internally_reserved":0,"allocated":536576,"data_stored":221578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59799999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":1.3660000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.44},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":1.492}]},{"osd":5,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.64700000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53600000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9940,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94361900,"statfs":{"total":96636764160,"available":96626585600,"internally_reserved":0,"allocated":1126400,"data_stored":811944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.6790000000000003,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.7270000000000003,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.56000000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.6239999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":7.6079999999999997,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.495},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.0209999999999999,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":9.7550000000000008,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.5489999999999999,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.36099999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.379},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45500000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.438},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:20:37.456 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-08T08:20:37.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-11-08T08:20:37.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.615+0000 7f8104800700 1 Processor -- start 2023-11-08T08:20:37.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.615+0000 7f8104800700 1 -- start start 2023-11-08T08:20:37.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f8104800700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f810006ba90 0x7f81000cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f8104800700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000cdfd0 con 0x7f8100159780 2023-11-08T08:20:37.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f8104800700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8100159b50 con 0x7f81000cf030 2023-11-08T08:20:37.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f8104800700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81000cf9e0 con 0x7f810006ba90 2023-11-08T08:20:37.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80ff59e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f810006ba90 0x7f81000cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80ff59e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f810006ba90 0x7f81000cf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41974/0 (socket says 172.21.15.116:41974) 2023-11-08T08:20:37.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80fed9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8100159780 0x7f810006d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52106/0 (socket says 172.21.15.116:52106) 2023-11-08T08:20:37.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80ff59e700 1 -- 172.21.15.116:0/1527439992 learned_addr learned my addr 172.21.15.116:0/1527439992 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:37.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80ff59e700 1 -- 172.21.15.116:0/1527439992 >> v1:172.21.15.129:6789/0 conn(0x7f81000cf030 legacy=0x7f810006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:37.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80ff59e700 1 -- 172.21.15.116:0/1527439992 >> v1:172.21.15.116:6789/0 conn(0x7f8100159780 legacy=0x7f810006d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f80ff59e700 1 -- 172.21.15.116:0/1527439992 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000ce270 con 0x7f810006ba90 2023-11-08T08:20:37.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.616+0000 7f80fdd9b700 1 -- 172.21.15.116:0/1527439992 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035586703 0 0) 0x7f81000cdfd0 con 0x7f8100159780 2023-11-08T08:20:37.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f80ff59e700 1 --2- 172.21.15.116:0/1527439992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f810006ba90 0x7f81000cf4a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009e50 tx=0x7f80f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5eb82c17a165a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:37.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f80fdd9b700 1 -- 172.21.15.116:0/1527439992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f800e070 con 0x7f810006ba90 2023-11-08T08:20:37.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f80fdd9b700 1 -- 172.21.15.116:0/1527439992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8003a20 con 0x7f810006ba90 2023-11-08T08:20:37.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f80fdd9b700 1 -- 172.21.15.116:0/1527439992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8003dd0 con 0x7f810006ba90 2023-11-08T08:20:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 -- 172.21.15.116:0/1527439992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f810006ba90 msgr2=0x7f81000cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 --2- 172.21.15.116:0/1527439992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f810006ba90 0x7f81000cf4a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009e50 tx=0x7f80f800b330 comp rx=0 tx=0).stop 2023-11-08T08:20:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 -- 172.21.15.116:0/1527439992 shutdown_connections 2023-11-08T08:20:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 --2- 172.21.15.116:0/1527439992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f810006ba90 0x7f81000cf4a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 -- 172.21.15.116:0/1527439992 >> 172.21.15.116:0/1527439992 conn(0x7f81000cc600 msgr2=0x7f81000cc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 -- 172.21.15.116:0/1527439992 shutdown_connections 2023-11-08T08:20:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.617+0000 7f8104800700 1 -- 172.21.15.116:0/1527439992 wait complete. 2023-11-08T08:20:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.619+0000 7f8104800700 1 Processor -- start 2023-11-08T08:20:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.619+0000 7f8104800700 1 -- start start 2023-11-08T08:20:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f810006ba90 0x7f8100146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81000cf030 0x7f81000aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 0x7f81000b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81000af4c0 con 0x7f8100159780 2023-11-08T08:20:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81000b1270 con 0x7f81000cf030 2023-11-08T08:20:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810014aab0 con 0x7f810006ba90 2023-11-08T08:20:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 0x7f81000b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 0x7f81000b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41984/0 (socket says 172.21.15.116:41984) 2023-11-08T08:20:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 -- 172.21.15.116:0/4245747906 learned_addr learned my addr 172.21.15.116:0/4245747906 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80fe59c700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81000cf030 0x7f81000aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80fed9d700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f810006ba90 0x7f8100146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 -- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f810006ba90 msgr2=0x7f8100146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f810006ba90 0x7f8100146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 -- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81000cf030 msgr2=0x7f81000aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81000cf030 0x7f81000aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 -- 172.21.15.116:0/4245747906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f8009b30 con 0x7f8100159780 2023-11-08T08:20:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80fe59c700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81000cf030 0x7f81000aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80fed9d700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f810006ba90 0x7f8100146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80ff59e700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 0x7f81000b0d30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8100071df0 tx=0x7f80f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004440 con 0x7f8100159780 2023-11-08T08:20:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f80045c0 con 0x7f8100159780 2023-11-08T08:20:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81000ce830 con 0x7f8100159780 2023-11-08T08:20:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004970 con 0x7f8100159780 2023-11-08T08:20:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.620+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81000ce550 con 0x7f8100159780 2023-11-08T08:20:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.621+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81000ceb10 con 0x7f8100159780 2023-11-08T08:20:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.622+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80f802d070 con 0x7f8100159780 2023-11-08T08:20:37.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.622+0000 7f80e77fe700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80ec0682f0 0x7f80ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:37.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.622+0000 7f80fed9d700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80ec0682f0 0x7f80ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:37.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.622+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f80f80876f0 con 0x7f8100159780 2023-11-08T08:20:37.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.623+0000 7f80fed9d700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80ec0682f0 0x7f80ec06a7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f80f4000f60 tx=0x7f80f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:37.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.627+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80f8054010 con 0x7f8100159780 2023-11-08T08:20:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.881+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f810014ac20 con 0x7f8100159780 2023-11-08T08:20:37.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.884+0000 7f80e77fe700 1 -- 172.21.15.116:0/4245747906 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 184+0+0 (secure 0 0 0) 0x7f80f8057630 con 0x7f8100159780 2023-11-08T08:20:37.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80ec0682f0 msgr2=0x7f80ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80ec0682f0 0x7f80ec06a7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f80f4000f60 tx=0x7f80f4009040 comp rx=0 tx=0).stop 2023-11-08T08:20:37.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 msgr2=0x7f81000b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:37.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 0x7f81000b0d30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8100071df0 tx=0x7f80f8004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 shutdown_connections 2023-11-08T08:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8100159780 0x7f81000b0d30 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f810006ba90 0x7f8100146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80ec0682f0 0x7f80ec06a7a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 --2- 172.21.15.116:0/4245747906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81000cf030 0x7f81000aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 >> 172.21.15.116:0/4245747906 conn(0x7f81000cc600 msgr2=0x7f8100155bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:37.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 shutdown_connections 2023-11-08T08:20:37.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:37.886+0000 7f8104800700 1 -- 172.21.15.116:0/4245747906 wait complete. 2023-11-08T08:20:37.890 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.0 2023-11-08T08:20:37.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.017+0000 7f6a5b920700 1 Processor -- start 2023-11-08T08:20:38.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.018+0000 7f6a5b920700 1 -- start start 2023-11-08T08:20:38.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.018+0000 7f6a5b920700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6a54071f70 0x7f6a540acb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.018+0000 7f6a5b920700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a540af0c0 con 0x7f6a540ae520 2023-11-08T08:20:38.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.018+0000 7f6a5b920700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a540b0590 con 0x7f6a5406a270 2023-11-08T08:20:38.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a5b920700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a540723f0 con 0x7f6a54071f70 2023-11-08T08:20:38.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a58ebb700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6a540ae520 0x7f6a540ac420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52116/0 (socket says 172.21.15.116:52116) 2023-11-08T08:20:38.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a58ebb700 1 -- 172.21.15.116:0/408285542 learned_addr learned my addr 172.21.15.116:0/408285542 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:38.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a59ebd700 1 --2- 172.21.15.116:0/408285542 >> v2:172.21.15.129:3300/0 conn(0x7f6a54071f70 0x7f6a540acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744497795 0 0) 0x7f6a540af0c0 con 0x7f6a540ae520 2023-11-08T08:20:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a400035b0 con 0x7f6a540ae520 2023-11-08T08:20:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1493868037 0 0) 0x7f6a400035b0 con 0x7f6a540ae520 2023-11-08T08:20:38.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 >> v2:172.21.15.129:3300/0 conn(0x7f6a54071f70 msgr2=0x7f6a540acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 --2- 172.21.15.116:0/408285542 >> v2:172.21.15.129:3300/0 conn(0x7f6a54071f70 0x7f6a540acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 >> v1:172.21.15.129:6789/0 conn(0x7f6a5406a270 legacy=0x7f6a5406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54159280 con 0x7f6a540ae520 2023-11-08T08:20:38.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f6a50001670 con 0x7f6a540ae520 2023-11-08T08:20:38.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 >> v1:172.21.15.116:6789/0 conn(0x7f6a540ae520 legacy=0x7f6a540ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a40003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a40004850 0x7f6a40004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a40005200 0x7f6a4000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540723f0 con 0x7f6a40003a00 2023-11-08T08:20:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4000c080 con 0x7f6a40005200 2023-11-08T08:20:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4000c240 con 0x7f6a40004850 2023-11-08T08:20:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a596bc700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a40003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a59ebd700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a40005200 0x7f6a4000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.019+0000 7f6a58ebb700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a40004850 0x7f6a40004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a596bc700 1 -- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a40004850 msgr2=0x7f6a40004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a596bc700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a40004850 0x7f6a40004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a596bc700 1 -- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a40005200 msgr2=0x7f6a4000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a596bc700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a40005200 0x7f6a4000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a59ebd700 1 -- 172.21.15.116:0/408285542 reap_dead start 2023-11-08T08:20:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a596bc700 1 -- 172.21.15.116:0/408285542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400035b0 con 0x7f6a40003a00 2023-11-08T08:20:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a596bc700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a40003e10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6a440017c0 tx=0x7f6a40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a361c2c940f1dde1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440052d0 con 0x7f6a40003a00 2023-11-08T08:20:38.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44005450 con 0x7f6a40003a00 2023-11-08T08:20:38.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a4bfff700 1 -- 172.21.15.116:0/408285542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440038c0 con 0x7f6a40003a00 2023-11-08T08:20:38.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a5b920700 1 -- 172.21.15.116:0/408285542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 msgr2=0x7f6a40003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a5b920700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a40003e10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6a440017c0 tx=0x7f6a40005880 comp rx=0 tx=0).stop 2023-11-08T08:20:38.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a5b920700 1 -- 172.21.15.116:0/408285542 shutdown_connections 2023-11-08T08:20:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a5b920700 1 --2- 172.21.15.116:0/408285542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a40003e10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.020+0000 7f6a5b920700 1 -- 172.21.15.116:0/408285542 >> 172.21.15.116:0/408285542 conn(0x7f6a540b2fc0 msgr2=0x7f6a54071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.021+0000 7f6a5b920700 1 -- 172.21.15.116:0/408285542 shutdown_connections 2023-11-08T08:20:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.021+0000 7f6a5b920700 1 -- 172.21.15.116:0/408285542 wait complete. 2023-11-08T08:20:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 Processor -- start 2023-11-08T08:20:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 -- start start 2023-11-08T08:20:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a5414e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a540c6520 0x7f6a540c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a540ad8b0 0x7f6a540c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5409cff0 con 0x7f6a40003a00 2023-11-08T08:20:38.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540c3b30 con 0x7f6a540c6520 2023-11-08T08:20:38.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a5b920700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540c7440 con 0x7f6a540ad8b0 2023-11-08T08:20:38.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.022+0000 7f6a596bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a5414e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a5414e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42000/0 (socket says 172.21.15.116:42000) 2023-11-08T08:20:38.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 -- 172.21.15.116:0/1476641086 learned_addr learned my addr 172.21.15.116:0/1476641086 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a58ebb700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a540c6520 0x7f6a540c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a59ebd700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a540ad8b0 0x7f6a540c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 -- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a540ad8b0 msgr2=0x7f6a540c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a540ad8b0 0x7f6a540c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 -- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a540c6520 msgr2=0x7f6a540c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a540c6520 0x7f6a540c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 -- 172.21.15.116:0/1476641086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54072670 con 0x7f6a40003a00 2023-11-08T08:20:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a58ebb700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a540c6520 0x7f6a540c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a59ebd700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a540ad8b0 0x7f6a540c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a596bc700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a5414e3d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6a44000c00 tx=0x7f6a44001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44001290 con 0x7f6a40003a00 2023-11-08T08:20:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44007040 con 0x7f6a40003a00 2023-11-08T08:20:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a540b0d10 con 0x7f6a40003a00 2023-11-08T08:20:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440073f0 con 0x7f6a40003a00 2023-11-08T08:20:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.023+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a540afc60 con 0x7f6a40003a00 2023-11-08T08:20:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.024+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a540743f0 con 0x7f6a40003a00 2023-11-08T08:20:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.025+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a4402d070 con 0x7f6a40003a00 2023-11-08T08:20:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.025+0000 7f6a49ffb700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6a380682f0 0x7f6a3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.025+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6a44086490 con 0x7f6a40003a00 2023-11-08T08:20:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.025+0000 7f6a58ebb700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6a380682f0 0x7f6a3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.026+0000 7f6a58ebb700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6a380682f0 0x7f6a3806a7a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6a540c6950 tx=0x7f6a50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.029+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a44052cf0 con 0x7f6a40003a00 2023-11-08T08:20:38.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.252+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a54071be0 con 0x7f6a40003a00 2023-11-08T08:20:38.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.252+0000 7f6a49ffb700 1 -- 172.21.15.116:0/1476641086 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f6a44056310 con 0x7f6a40003a00 2023-11-08T08:20:38.254 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:38.254 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:20:38.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6a380682f0 msgr2=0x7f6a3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6a380682f0 0x7f6a3806a7a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6a540c6950 tx=0x7f6a50009040 comp rx=0 tx=0).stop 2023-11-08T08:20:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 msgr2=0x7f6a5414e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a5414e3d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6a44000c00 tx=0x7f6a44001120 comp rx=0 tx=0).stop 2023-11-08T08:20:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 shutdown_connections 2023-11-08T08:20:38.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6a540ad8b0 0x7f6a540c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40003a00 0x7f6a5414e3d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6a380682f0 0x7f6a3806a7a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 --2- 172.21.15.116:0/1476641086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a540c6520 0x7f6a540c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 >> 172.21.15.116:0/1476641086 conn(0x7f6a540b2fc0 msgr2=0x7f6a5406d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 shutdown_connections 2023-11-08T08:20:38.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.254+0000 7f6a5b920700 1 -- 172.21.15.116:0/1476641086 wait complete. 2023-11-08T08:20:38.258 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:20:38.614 INFO:tasks.ceph.mds.b.smithi129.stderr:2023-11-08T08:20:38.614+0000 7f9ff5884700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:20:38.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-08T08:20:33.012833+0000', 'modified': '2023-11-08T08:20:36.023322+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5781': {'gid': 5781, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/409645101', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 409645101}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 409645101}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-08T08:20:38.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-08T08:20:38.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-08T08:20:38.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.792+0000 7f7cea432700 1 Processor -- start 2023-11-08T08:20:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.792+0000 7f7cea432700 1 -- start start 2023-11-08T08:20:38.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.793+0000 7f7cea432700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 0x7f7ce4160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.793+0000 7f7cea432700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 0x7f7ce406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.793+0000 7f7cea432700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce41607a0 con 0x7f7ce406cdc0 2023-11-08T08:20:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.793+0000 7f7cea432700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce415cbe0 con 0x7f7ce40718e0 2023-11-08T08:20:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.793+0000 7f7cea432700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4160a50 con 0x7f7ce415c810 2023-11-08T08:20:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 0x7f7ce4160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce37fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 0x7f7ce406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 0x7f7ce4160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:40100/0 (socket says 172.21.15.116:40100) 2023-11-08T08:20:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce89cf700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7ce406cdc0 0x7f7ce406a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:39810/0 (socket says 172.21.15.116:39810) 2023-11-08T08:20:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce37fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 0x7f7ce406c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:49920/0 (socket says 172.21.15.116:49920) 2023-11-08T08:20:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 -- 172.21.15.116:0/1145084614 learned_addr learned my addr 172.21.15.116:0/1145084614 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 -- 172.21.15.116:0/1145084614 >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 msgr2=0x7f7ce406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/1145084614 >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 0x7f7ce406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 -- 172.21.15.116:0/1145084614 >> v1:172.21.15.129:6789/0 conn(0x7f7ce406cdc0 legacy=0x7f7ce406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 -- 172.21.15.116:0/1145084614 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4160d30 con 0x7f7ce415c810 2023-11-08T08:20:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/1145084614 >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 0x7f7ce406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/1145084614 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 0x7f7ce4160260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80098f0 tx=0x7f7cd8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=827b54a753fa3ea8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce2ffd700 1 -- 172.21.15.116:0/1145084614 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8011830 con 0x7f7ce415c810 2023-11-08T08:20:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce2ffd700 1 -- 172.21.15.116:0/1145084614 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8011990 con 0x7f7ce415c810 2023-11-08T08:20:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.794+0000 7f7ce2ffd700 1 -- 172.21.15.116:0/1145084614 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8011d20 con 0x7f7ce415c810 2023-11-08T08:20:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 -- 172.21.15.116:0/1145084614 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 msgr2=0x7f7ce4160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 --2- 172.21.15.116:0/1145084614 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 0x7f7ce4160260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80098f0 tx=0x7f7cd8009c00 comp rx=0 tx=0).stop 2023-11-08T08:20:38.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 -- 172.21.15.116:0/1145084614 shutdown_connections 2023-11-08T08:20:38.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 --2- 172.21.15.116:0/1145084614 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce415c810 0x7f7ce4160260 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 --2- 172.21.15.116:0/1145084614 >> v2:172.21.15.129:3300/0 conn(0x7f7ce40718e0 0x7f7ce406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 -- 172.21.15.116:0/1145084614 >> 172.21.15.116:0/1145084614 conn(0x7f7ce406c010 msgr2=0x7f7ce40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 -- 172.21.15.116:0/1145084614 shutdown_connections 2023-11-08T08:20:38.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.795+0000 7f7cea432700 1 -- 172.21.15.116:0/1145084614 wait complete. 2023-11-08T08:20:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.796+0000 7f7cea432700 1 Processor -- start 2023-11-08T08:20:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.796+0000 7f7cea432700 1 -- start start 2023-11-08T08:20:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7cea432700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce406cdc0 0x7f7ce415e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7cea432700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 0x7f7ce415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7cea432700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce415c810 0x7f7ce415f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7cea432700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce415e8c0 con 0x7f7ce40718e0 2023-11-08T08:20:38.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7cea432700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce415d2e0 con 0x7f7ce415c810 2023-11-08T08:20:38.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7cea432700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce415d5c0 con 0x7f7ce406cdc0 2023-11-08T08:20:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 0x7f7ce415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 0x7f7ce415a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42008/0 (socket says 172.21.15.116:42008) 2023-11-08T08:20:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 -- 172.21.15.116:0/313515518 learned_addr learned my addr 172.21.15.116:0/313515518 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce89cf700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce415c810 0x7f7ce415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce406cdc0 0x7f7ce415e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 -- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce406cdc0 msgr2=0x7f7ce415e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce406cdc0 0x7f7ce415e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 -- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce415c810 msgr2=0x7f7ce415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce415c810 0x7f7ce415f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce37fe700 1 -- 172.21.15.116:0/313515518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd40094a0 con 0x7f7ce40718e0 2023-11-08T08:20:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce89cf700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce415c810 0x7f7ce415f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.797+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce406cdc0 0x7f7ce415e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.798+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 0x7f7ce415a530 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400ebe0 tx=0x7f7cd4009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.798+0000 7f7ce0ff9700 1 -- 172.21.15.116:0/313515518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400c910 con 0x7f7ce40718e0 2023-11-08T08:20:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.798+0000 7f7ce0ff9700 1 -- 172.21.15.116:0/313515518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd400ca70 con 0x7f7ce40718e0 2023-11-08T08:20:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.798+0000 7f7ce0ff9700 1 -- 172.21.15.116:0/313515518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400cde0 con 0x7f7ce40718e0 2023-11-08T08:20:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.798+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd8009550 con 0x7f7ce40718e0 2023-11-08T08:20:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.798+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce40ceb20 con 0x7f7ce40718e0 2023-11-08T08:20:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.799+0000 7f7ce0ff9700 1 -- 172.21.15.116:0/313515518 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7cd4019030 con 0x7f7ce40718e0 2023-11-08T08:20:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.799+0000 7f7ce0ff9700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7cd00682f0 0x7f7cd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.799+0000 7f7ce0ff9700 1 -- 172.21.15.116:0/313515518 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7cd4085cd0 con 0x7f7ce40718e0 2023-11-08T08:20:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.799+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7cd00682f0 0x7f7cd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.800+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7cd00682f0 0x7f7cd006a7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009c40 tx=0x7f7cd8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.801+0000 7f7cf65d1b80 1 Processor -- start 2023-11-08T08:20:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.801+0000 7f7cf65d1b80 1 -- start start 2023-11-08T08:20:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 0x55e3c028c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c028db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c020ade0 con 0x55e3c02d2f10 2023-11-08T08:20:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c0177140 con 0x55e3c02b5620 2023-11-08T08:20:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3c01e5db0 con 0x55e3c02c9a80 2023-11-08T08:20:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 0x55e3c028c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 0x55e3c028c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42016/0 (socket says 172.21.15.116:42016) 2023-11-08T08:20:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 -- 172.21.15.116:0/569876803 learned_addr learned my addr 172.21.15.116:0/569876803 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce89cf700 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c028db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 -- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 msgr2=0x55e3c02d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 -- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 msgr2=0x55e3c028db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c028db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.802+0000 7f7ce3fff700 1 -- 172.21.15.116:0/569876803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3bffd3790 con 0x55e3c02d2f10 2023-11-08T08:20:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 0x55e3c028c9b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8011ca0 tx=0x7f7cd8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5044d8006eab476 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cc9ffb700 1 -- 172.21.15.116:0/569876803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8003960 con 0x55e3c02d2f10 2023-11-08T08:20:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cc9ffb700 1 -- 172.21.15.116:0/569876803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8003ac0 con 0x55e3c02d2f10 2023-11-08T08:20:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cc9ffb700 1 -- 172.21.15.116:0/569876803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8003e30 con 0x55e3c02d2f10 2023-11-08T08:20:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/569876803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 msgr2=0x55e3c028c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 0x55e3c028c9b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8011ca0 tx=0x7f7cd8010000 comp rx=0 tx=0).stop 2023-11-08T08:20:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/569876803 shutdown_connections 2023-11-08T08:20:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c028db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02d2f10 0x55e3c028c9b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/569876803 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/569876803 >> 172.21.15.116:0/569876803 conn(0x55e3c0282ce0 msgr2=0x55e3c028bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/569876803 shutdown_connections 2023-11-08T08:20:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.803+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/569876803 wait complete. 2023-11-08T08:20:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.804+0000 7f7cf65d1b80 1 -- start start 2023-11-08T08:20:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.804+0000 7f7cf65d1b80 1 Processor -- start 2023-11-08T08:20:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 0x55e3c02d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c01df120 con 0x55e3c02b5620 2023-11-08T08:20:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bffd4390 con 0x55e3c02d2f10 2023-11-08T08:20:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3c0177340 con 0x55e3c02c9a80 2023-11-08T08:20:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cf65d1b80 10 client.0 _ll_register_callbacks cb 0x55e3c017d400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:20:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 0x55e3c02d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 0x55e3c02d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42028/0 (socket says 172.21.15.116:42028) 2023-11-08T08:20:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 -- 172.21.15.116:0/3835915299 learned_addr learned my addr 172.21.15.116:0/3835915299 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce89cf700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 -- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 msgr2=0x55e3c02d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 -- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 msgr2=0x55e3c02c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 -- 172.21.15.116:0/3835915299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd8022070 con 0x55e3c02b5620 2023-11-08T08:20:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 0x55e3c02d11f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009130 tx=0x7f7cd80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76274f3bc4ea5cc9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.805+0000 7f7cc2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:20:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/3835915299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd801c640 con 0x55e3c02b5620 2023-11-08T08:20:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/3835915299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7cd8038070 con 0x55e3c02b5620 2023-11-08T08:20:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3c01e3030 con 0x55e3c02b5620 2023-11-08T08:20:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/3835915299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd801c7a0 con 0x55e3c02b5620 2023-11-08T08:20:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 10 client.5904 fetch_fsmap learned FSMap version 24 2023-11-08T08:20:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55e3c00c00a0 con 0x55e3c02b5620 2023-11-08T08:20:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/3835915299 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e3c01e3030 con 0x55e3c02b5620 2023-11-08T08:20:38.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/3835915299 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f7cd802f020 con 0x55e3c02b5620 2023-11-08T08:20:38.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 10 client.5904 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:20:38.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 10 client.5904 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-08T08:20:38.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 20 client.5904 populate_metadata read hostname 'smithi116' 2023-11-08T08:20:38.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02ca430 0x55e3c02ca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 4 client.5904 mds_command: new command op to 5781 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:20:38.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3c002eef0 con 0x55e3c02ca430 2023-11-08T08:20:38.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.806+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02ca430 0x55e3c02ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cc2ffd700 10 client.5904 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:38.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02ca430 0x55e3c02ca800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a7553c81006a2e server_cookie=933cb758a161aba7 in_seq=0 out_seq=0 2023-11-08T08:20:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/3835915299 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e3c002eef0 con 0x55e3c02ca430 2023-11-08T08:20:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cc2ffd700 10 client.5904 handle_command_reply: tid=0 2023-11-08T08:20:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 client.5904 shutdown 2023-11-08T08:20:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 msgr2=0x55e3c02d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 0x55e3c02d11f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009130 tx=0x7f7cd80047c0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 shutdown_connections 2023-11-08T08:20:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02b5620 0x55e3c02d11f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02ca430 0x55e3c02ca800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/3835915299 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02c9a80 0x55e3c02d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.807+0000 7f7cc2ffd700 0 client.5904 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.808+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 >> 172.21.15.116:0/3835915299 conn(0x55e3c0282ce0 msgr2=0x55e3c028bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.808+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 shutdown_connections 2023-11-08T08:20:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.808+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/3835915299 wait complete. 2023-11-08T08:20:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.808+0000 7f7cf65d1b80 20 client.5904 trim_cache size 0 max 16384 2023-11-08T08:20:38.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 Processor -- start 2023-11-08T08:20:38.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 -- start start 2023-11-08T08:20:38.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c02cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02d2f10 0x55e3c02d2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c0288cc0 con 0x55e3c02c9a80 2023-11-08T08:20:38.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c02800a0 con 0x55e3c02b5620 2023-11-08T08:20:38.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3c02e1dd0 con 0x55e3c02d2f10 2023-11-08T08:20:38.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42038/0 (socket says 172.21.15.116:42038) 2023-11-08T08:20:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.825+0000 7f7ce37fe700 1 -- 172.21.15.116:0/4085676924 learned_addr learned my addr 172.21.15.116:0/4085676924 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7ce37fe700 1 -- 172.21.15.116:0/4085676924 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02d2f10 msgr2=0x55e3c02d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02d2f10 0x55e3c02d2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7ce37fe700 1 -- 172.21.15.116:0/4085676924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 msgr2=0x55e3c02cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c02cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7ce37fe700 1 -- 172.21.15.116:0/4085676924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3c01a62e0 con 0x55e3c02c9a80 2023-11-08T08:20:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02c89f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40526f0 tx=0x7f7cd4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f52bb0f03a3732d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cca7fc700 1 -- 172.21.15.116:0/4085676924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4082070 con 0x55e3c02c9a80 2023-11-08T08:20:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cca7fc700 1 -- 172.21.15.116:0/4085676924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd406d070 con 0x55e3c02c9a80 2023-11-08T08:20:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cca7fc700 1 -- 172.21.15.116:0/4085676924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40558e0 con 0x55e3c02c9a80 2023-11-08T08:20:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/4085676924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 msgr2=0x55e3c02c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02c89f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40526f0 tx=0x7f7cd4000bc0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/4085676924 shutdown_connections 2023-11-08T08:20:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02d2f10 0x55e3c02d2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02b5620 0x55e3c02cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/4085676924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02c89f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/4085676924 >> 172.21.15.116:0/4085676924 conn(0x55e3c0282ce0 msgr2=0x55e3c028be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/4085676924 shutdown_connections 2023-11-08T08:20:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.826+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/4085676924 wait complete. 2023-11-08T08:20:38.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.827+0000 7f7cf65d1b80 1 -- start start 2023-11-08T08:20:38.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.827+0000 7f7cf65d1b80 1 Processor -- start 2023-11-08T08:20:38.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02b5620 0x55e3c032b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02bae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c032bc40 con 0x55e3c02c9a80 2023-11-08T08:20:38.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c032d860 con 0x55e3c02d2f10 2023-11-08T08:20:38.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3c02afe70 con 0x55e3c02b5620 2023-11-08T08:20:38.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 10 client.0 _ll_register_callbacks cb 0x55e3c017d400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:20:38.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02bae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02bae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42052/0 (socket says 172.21.15.116:42052) 2023-11-08T08:20:38.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 -- 172.21.15.116:0/546738673 learned_addr learned my addr 172.21.15.116:0/546738673 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:38.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 -- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02b5620 msgr2=0x55e3c032b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02b5620 0x55e3c032b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 -- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 msgr2=0x55e3c02d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 -- 172.21.15.116:0/546738673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce40ceb20 con 0x55e3c02c9a80 2023-11-08T08:20:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7ce37fe700 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02bae40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4054540 tx=0x7f7cd801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58aa662c538bafe9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cc2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:20:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/546738673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4082040 con 0x55e3c02c9a80 2023-11-08T08:20:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/546738673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7cd406d3e0 con 0x55e3c02c9a80 2023-11-08T08:20:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.828+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cd8003ac0 con 0x55e3c02c9a80 2023-11-08T08:20:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/546738673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd4051fe0 con 0x55e3c02c9a80 2023-11-08T08:20:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 10 client.5910 fetch_fsmap learned FSMap version 24 2023-11-08T08:20:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55e3c00341d0 con 0x55e3c02c9a80 2023-11-08T08:20:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/546738673 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8003ac0 con 0x55e3c02c9a80 2023-11-08T08:20:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/546738673 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f7cd4052140 con 0x55e3c02c9a80 2023-11-08T08:20:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 10 client.5910 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:20:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 10 client.5910 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-08T08:20:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 20 client.5910 populate_metadata read hostname 'smithi116' 2023-11-08T08:20:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02e42e0 0x55e3c02e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 4 client.5910 mds_command: new command op to 5781 tid=0[{"prefix": "status"}] 2023-11-08T08:20:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e3c0064040 con 0x55e3c02e42e0 2023-11-08T08:20:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02e42e0 0x55e3c02e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7cc2ffd700 10 client.5910 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.829+0000 7f7ce3fff700 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02e42e0 0x55e3c02e46d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f84b126c303f552 server_cookie=2f55275e41c3b26 in_seq=0 out_seq=0 2023-11-08T08:20:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cc2ffd700 1 -- 172.21.15.116:0/546738673 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e3c0064040 con 0x55e3c02e42e0 2023-11-08T08:20:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cc2ffd700 10 client.5910 handle_command_reply: tid=0 2023-11-08T08:20:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 client.5910 shutdown 2023-11-08T08:20:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 msgr2=0x55e3c02bae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02bae40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4054540 tx=0x7f7cd801e040 comp rx=0 tx=0).stop 2023-11-08T08:20:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 shutdown_connections 2023-11-08T08:20:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e3c02d2f10 0x55e3c02d2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55e3c02e42e0 0x55e3c02e46d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55e3c02b5620 0x55e3c032b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cc2ffd700 0 client.5910 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 --2- 172.21.15.116:0/546738673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3c02c9a80 0x55e3c02bae40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 >> 172.21.15.116:0/546738673 conn(0x55e3c0282ce0 msgr2=0x55e3c02ca800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 shutdown_connections 2023-11-08T08:20:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 1 -- 172.21.15.116:0/546738673 wait complete. 2023-11-08T08:20:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cf65d1b80 20 client.5910 trim_cache size 0 max 16384 2023-11-08T08:20:38.857 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:20:38.857 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "2f470591-d661-4b50-b7a5-062dfcf46b1d", 2023-11-08T08:20:38.857 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-08T08:20:38.857 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5781, 2023-11-08T08:20:38.857 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-08T08:20:38.858 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-08T08:20:38.858 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-08T08:20:38.858 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 5.8163022529999999, 2023-11-08T08:20:38.858 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2023-11-08T08:20:38.858 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 44, 2023-11-08T08:20:38.859 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 44, 2023-11-08T08:20:38.859 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 11.144630606 2023-11-08T08:20:38.859 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:20:38.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7cd00682f0 msgr2=0x7f7cd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7cd00682f0 0x7f7cd006a7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009c40 tx=0x7f7cd8008030 comp rx=0 tx=0).stop 2023-11-08T08:20:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 msgr2=0x7f7ce415a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 0x7f7ce415a530 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400ebe0 tx=0x7f7cd4009ec0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 shutdown_connections 2023-11-08T08:20:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f7cd00682f0 0x7f7cd006a7a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce415c810 0x7f7ce415f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ce406cdc0 0x7f7ce415e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 --2- 172.21.15.116:0/313515518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce40718e0 0x7f7ce415a530 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.830+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 >> 172.21.15.116:0/313515518 conn(0x7f7ce406c010 msgr2=0x7f7ce406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.831+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 shutdown_connections 2023-11-08T08:20:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.831+0000 7f7cea432700 1 -- 172.21.15.116:0/313515518 wait complete. 2023-11-08T08:20:38.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:20:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.982+0000 7f9b6080e700 1 Processor -- start 2023-11-08T08:20:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.982+0000 7f9b6080e700 1 -- start start 2023-11-08T08:20:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b6080e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b5c06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b6080e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5c06ab80 con 0x7f9b5c15cc80 2023-11-08T08:20:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b6080e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5c0b1810 con 0x7f9b5c06e570 2023-11-08T08:20:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b6080e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b2490 con 0x7f9b5c06a270 2023-11-08T08:20:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b59d9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9b5c15cc80 0x7f9b5c06de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52124/0 (socket says 172.21.15.116:52124) 2023-11-08T08:20:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b59d9b700 1 -- 172.21.15.116:0/3629645620 learned_addr learned my addr 172.21.15.116:0/3629645620 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b5a59c700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b5c06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608979762 0 0) 0x7f9b5c06ab80 con 0x7f9b5c15cc80 2023-11-08T08:20:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b440035b0 con 0x7f9b5c15cc80 2023-11-08T08:20:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.983+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1101586031 0 0) 0x7f9b440035b0 con 0x7f9b5c15cc80 2023-11-08T08:20:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 msgr2=0x7f9b5c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b5c06a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 >> v1:172.21.15.129:6789/0 conn(0x7f9b5c06e570 legacy=0x7f9b5c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0b1af0 con 0x7f9b5c15cc80 2023-11-08T08:20:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9b4c001670 con 0x7f9b5c15cc80 2023-11-08T08:20:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 >> v1:172.21.15.116:6789/0 conn(0x7f9b5c15cc80 legacy=0x7f9b5c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b44003a30 0x7f9b44003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b44004850 0x7f9b44004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b440052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b2490 con 0x7f9b44004850 2023-11-08T08:20:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4400bbd0 con 0x7f9b44003a30 2023-11-08T08:20:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4400bd90 con 0x7f9b5c06a270 2023-11-08T08:20:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b44004850 0x7f9b44004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5a59c700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b44003a30 0x7f9b44003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5ad9d700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b440052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 -- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 msgr2=0x7f9b440052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b440052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 -- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b44003a30 msgr2=0x7f9b44003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b44003a30 0x7f9b44003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 -- 172.21.15.116:0/3629645620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440035b0 con 0x7f9b44004850 2023-11-08T08:20:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5ad9d700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b5c06a270 0x7f9b440052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b59d9b700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b44004850 0x7f9b44004cc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00c270 tx=0x7f9b44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892d2d39c537615 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c007c10 con 0x7f9b44004850 2023-11-08T08:20:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4c00cd90 con 0x7f9b44004850 2023-11-08T08:20:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.984+0000 7f9b5959a700 1 -- 172.21.15.116:0/3629645620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c005ba0 con 0x7f9b44004850 2023-11-08T08:20:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b6080e700 1 -- 172.21.15.116:0/3629645620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b44004850 msgr2=0x7f9b44004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b6080e700 1 --2- 172.21.15.116:0/3629645620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b44004850 0x7f9b44004cc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00c270 tx=0x7f9b44005880 comp rx=0 tx=0).stop 2023-11-08T08:20:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b5ad9d700 1 -- 172.21.15.116:0/3629645620 reap_dead start 2023-11-08T08:20:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b6080e700 1 -- 172.21.15.116:0/3629645620 shutdown_connections 2023-11-08T08:20:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b6080e700 1 -- 172.21.15.116:0/3629645620 >> 172.21.15.116:0/3629645620 conn(0x7f9b5c06c010 msgr2=0x7f9b5c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b6080e700 1 -- 172.21.15.116:0/3629645620 shutdown_connections 2023-11-08T08:20:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.985+0000 7f9b6080e700 1 -- 172.21.15.116:0/3629645620 wait complete. 2023-11-08T08:20:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.986+0000 7f9b6080e700 1 Processor -- start 2023-11-08T08:20:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.986+0000 7f9b6080e700 1 -- start start 2023-11-08T08:20:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b6080e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b44004850 0x7f9b5c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b6080e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b5c0b4700 0x7f9b5c151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b6080e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 0x7f9b5c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b6080e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c06e9b0 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b6080e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b58f0 con 0x7f9b5c0b4700 2023-11-08T08:20:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b6080e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b3a90 con 0x7f9b44004850 2023-11-08T08:20:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 0x7f9b5c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 0x7f9b5c0b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42070/0 (socket says 172.21.15.116:42070) 2023-11-08T08:20:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 -- 172.21.15.116:0/1856830090 learned_addr learned my addr 172.21.15.116:0/1856830090 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5a59c700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b44004850 0x7f9b5c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b59d9b700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b5c0b4700 0x7f9b5c151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 -- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b44004850 msgr2=0x7f9b5c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b44004850 0x7f9b5c0cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 -- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b5c0b4700 msgr2=0x7f9b5c151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b5c0b4700 0x7f9b5c151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5ad9d700 1 -- 172.21.15.116:0/1856830090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0b20b0 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b5a59c700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b44004850 0x7f9b5c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.987+0000 7f9b59d9b700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b5c0b4700 0x7f9b5c151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.988+0000 7f9b5ad9d700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 0x7f9b5c0b3550 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c0c6d50 tx=0x7f9b5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.988+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54003c10 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.988+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b54003d90 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.988+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54005310 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.988+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c0b1dd0 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.988+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c15daa0 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.989+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c152430 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.989+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9b54005470 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.989+0000 7f9b4affd700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9b3c0682f0 0x7f9b3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.989+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9b54085e00 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.990+0000 7f9b5a59c700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9b3c0682f0 0x7f9b3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.990+0000 7f9b5a59c700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9b3c0682f0 0x7f9b3c06a7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009b50 tx=0x7f9b50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:38.994+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9b5c152430 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.218+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9b5c0b5a30 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.218+0000 7f9b4affd700 1 -- 172.21.15.116:0/1856830090 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9b5c0b5a30 con 0x7f9b5c0b30e0 2023-11-08T08:20:39.219 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:20:39.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9b3c0682f0 msgr2=0x7f9b3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:39.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9b3c0682f0 0x7f9b3c06a7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009b50 tx=0x7f9b50008040 comp rx=0 tx=0).stop 2023-11-08T08:20:39.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 msgr2=0x7f9b5c0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:39.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 0x7f9b5c0b3550 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c0c6d50 tx=0x7f9b5400ee70 comp rx=0 tx=0).stop 2023-11-08T08:20:39.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 shutdown_connections 2023-11-08T08:20:39.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b5c0b30e0 0x7f9b5c0b3550 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:39.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9b44004850 0x7f9b5c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9b3c0682f0 0x7f9b3c06a7a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 --2- 172.21.15.116:0/1856830090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b5c0b4700 0x7f9b5c151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 >> 172.21.15.116:0/1856830090 conn(0x7f9b5c06c010 msgr2=0x7f9b5c153200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 shutdown_connections 2023-11-08T08:20:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:39.220+0000 7f9b6080e700 1 -- 172.21.15.116:0/1856830090 wait complete. 2023-11-08T08:20:39.233 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:20:39.233 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:20:39.234 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-08T08:20:39.234 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-08T08:20:39.234 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-08T08:20:39.235 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:20:39.235 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-08T08:20:39.235 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:20:39.235 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:20:39.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.236 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-08T08:20:39.255 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:20:39.255 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:20:39.255 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:20:39.256 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.256 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:20:39.256 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.256 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:20:39.256 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:20:39.257 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-08T08:20:39.257 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:20:39.257 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:20:39.257 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-08T08:20:39.257 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:20:39.258 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:20:39.258 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:20:39.258 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:20:39.258 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:20:39.258 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42178sec preferred_lft 42178sec 2023-11-08T08:20:39.259 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-08T08:20:39.259 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.259 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:20:39.259 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:20:39.259 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-08T08:20:39.260 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.260 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::8c0:d4ff:fe3b:7f6b/64 scope link 2023-11-08T08:20:39.260 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.260 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:20:39.260 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-08T08:20:39.261 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bc76:a3ff:fe22:41c7/64 scope link 2023-11-08T08:20:39.261 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.261 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:20:39.261 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:ab:30:29:e8:99 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-08T08:20:39.261 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bcab:30ff:fe29:e899/64 scope link 2023-11-08T08:20:39.261 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:20:39.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.262 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:20:39.320 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-08T08:20:39.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.322 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-08T08:20:39.381 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-08T08:20:39.382 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:20:39.382 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:20:39.382 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:20:39.382 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:20:39.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.383 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:20:39.384 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:20:39.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:20:39.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:20:39.385 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:20:39.385 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:20:39.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.486 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:20:39.504 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-08T08:20:39.504 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-08T08:20:39.505 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:20:39.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-08T08:20:39.571 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-08T08:20:39.572 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:20:39.573 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-08T08:20:39.630 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:20:39.630 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:20:39.631 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:20:39.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:20:39.757 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:20:39.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:20:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:20:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:20:39.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:39.826 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:20:39.883 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-08T08:20:39.884 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-11-08T08:20:39.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:20:39.952 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:20:39.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:20:40.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-08T08:20:40.011+0000 7f35f6a075c0 -1 init, newargv = 0x55f1c5643880 newargc=15 2023-11-08T08:20:40.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[142354]: starting ceph client 2023-11-08T08:20:40.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[142354]: starting fuse 2023-11-08T08:20:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:20:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:20:40.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:40.024 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:20:40.081 INFO:teuthology.orchestra.run.smithi116.stdout:46 2023-11-08T08:20:40.082 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46] 2023-11-08T08:20:40.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:20:40.083 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-08T08:20:40.083 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-08T08:20:40.084 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:20:40.084 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-08T08:20:40.084 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:20:40.085 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-08T08:20:40.085 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-08T08:20:40.085 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-08T08:20:40.086 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-08T08:20:40.086 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:20:40.087 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-08T08:20:40.087 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-08T08:20:40.087 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-08T08:20:40.088 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-08T08:20:40.088 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:20:40.088 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-08T08:20:40.089 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:20:40.089 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:20:40.090 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:20:40.090 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-08T08:20:40.090 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-08T08:20:40.091 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-08T08:20:40.091 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:20:40.094 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:20:40.095 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-08T08:20:40.095 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:20:40.203 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.142354.asok 2023-11-08T08:20:40.386 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.142354.asok 2023-11-08T08:20:40.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:40.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.142354.asok status 2023-11-08T08:20:40.541 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:20:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-08T08:20:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:20:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:20:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-08T08:20:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-08T08:20:40.543 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-08T08:20:40.543 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "142354", 2023-11-08T08:20:40.543 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-08T08:20:40.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:40.543 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-08T08:20:40.544 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-08T08:20:40.544 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5922, 2023-11-08T08:20:40.544 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-08T08:20:40.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-08T08:20:40.544 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-08T08:20:40.545 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5922 2023-11-08T08:20:40.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:40.545 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:20:40.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:20:40.546 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:20:40.546 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3692035600 2023-11-08T08:20:40.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:40.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:40.546 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:20:40.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:20:40.547 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:20:40.547 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3692035600 2023-11-08T08:20:40.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:40.547 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.5922 192.168.0.1:0/3692035600", 2023-11-08T08:20:40.548 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/3692035600", 2023-11-08T08:20:40.548 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-08T08:20:40.548 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 24, 2023-11-08T08:20:40.548 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 44, 2023-11-08T08:20:40.549 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:20:40.549 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-08T08:20:40.549 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-08T08:20:40.549 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:20:40.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:40.551 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-08T08:20:40.552 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-08T08:20:40.570 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-08T08:20:40.570 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-08T08:20:40.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:40.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:20:40.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:40.638 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2023-11-08T08:20:40.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:20:40.703 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2023-11-08T08:20:40.764 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:20:40.764 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:20:40.764 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:20:40.765 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:20:40.765 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2023-11-08T08:20:40.766 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:20:40.871 INFO:teuthology.orchestra.run.smithi116.stdout:1099511627777 2023-11-08T08:20:41.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:20:41.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.218+0000 7ff5f9ad0700 1 Processor -- start 2023-11-08T08:20:41.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.218+0000 7ff5f9ad0700 1 -- start start 2023-11-08T08:20:41.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f9ad0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5f406e400 0x7ff5f406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f9ad0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f415e010 0x7ff5f40ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f9ad0700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f415af30 con 0x7ff5f40d15f0 2023-11-08T08:20:41.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f9ad0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f415a9d0 con 0x7ff5f406e400 2023-11-08T08:20:41.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f9ad0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f41584e0 con 0x7ff5f415e010 2023-11-08T08:20:41.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f37fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5f406e400 0x7ff5f406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f2ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff5f40d15f0 0x7ff5f415b3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52130/0 (socket says 172.21.15.116:52130) 2023-11-08T08:20:41.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.219+0000 7ff5f2ffd700 1 -- 172.21.15.116:0/4090486082 learned_addr learned my addr 172.21.15.116:0/4090486082 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:41.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f3fff700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f415e010 0x7ff5f40ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f37fe700 1 -- 172.21.15.116:0/4090486082 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f415e010 msgr2=0x7ff5f40ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f37fe700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f415e010 0x7ff5f40ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f37fe700 1 -- 172.21.15.116:0/4090486082 >> v1:172.21.15.116:6789/0 conn(0x7ff5f40d15f0 legacy=0x7ff5f415b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f37fe700 1 -- 172.21.15.116:0/4090486082 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f415ef40 con 0x7ff5f406e400 2023-11-08T08:20:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f27fc700 1 -- 172.21.15.116:0/4090486082 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011780651 0 0) 0x7ff5f415af30 con 0x7ff5f40d15f0 2023-11-08T08:20:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f3fff700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f415e010 0x7ff5f40ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:20:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f37fe700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f406e400 0x7ff5f406e7d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009840 tx=0x7ff5e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c699018db767a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f27fc700 1 -- 172.21.15.116:0/4090486082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4011830 con 0x7ff5f406e400 2023-11-08T08:20:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f27fc700 1 -- 172.21.15.116:0/4090486082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e4011990 con 0x7ff5f406e400 2023-11-08T08:20:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.220+0000 7ff5f27fc700 1 -- 172.21.15.116:0/4090486082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4011d20 con 0x7ff5f406e400 2023-11-08T08:20:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/4090486082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f406e400 msgr2=0x7ff5f406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f406e400 0x7ff5f406e7d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009840 tx=0x7ff5e4009b50 comp rx=0 tx=0).stop 2023-11-08T08:20:41.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/4090486082 shutdown_connections 2023-11-08T08:20:41.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f406e400 0x7ff5f406e7d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/4090486082 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f415e010 0x7ff5f40ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/4090486082 >> 172.21.15.116:0/4090486082 conn(0x7ff5f406a270 msgr2=0x7ff5f406a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:41.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/4090486082 shutdown_connections 2023-11-08T08:20:41.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.221+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/4090486082 wait complete. 2023-11-08T08:20:41.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.222+0000 7ff5f9ad0700 1 Processor -- start 2023-11-08T08:20:41.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.222+0000 7ff5f9ad0700 1 -- start start 2023-11-08T08:20:41.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f406e400 0x7ff5f413f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 0x7ff5f415cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5f415e010 0x7ff5f415d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cb630 con 0x7ff5f40d15f0 2023-11-08T08:20:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cb8d0 con 0x7ff5f415e010 2023-11-08T08:20:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cbbb0 con 0x7ff5f406e400 2023-11-08T08:20:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 0x7ff5f415cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 0x7ff5f415cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42086/0 (socket says 172.21.15.116:42086) 2023-11-08T08:20:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 -- 172.21.15.116:0/3414132371 learned_addr learned my addr 172.21.15.116:0/3414132371 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f37fe700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f406e400 0x7ff5f413f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f3fff700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5f415e010 0x7ff5f415d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 -- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f406e400 msgr2=0x7ff5f413f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f406e400 0x7ff5f413f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 -- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5f415e010 msgr2=0x7ff5f415d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5f415e010 0x7ff5f415d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 -- 172.21.15.116:0/3414132371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e40094a0 con 0x7ff5f40d15f0 2023-11-08T08:20:41.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f2ffd700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 0x7ff5f415cbb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00c540 tx=0x7ff5dc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc004c80 con 0x7ff5f40d15f0 2023-11-08T08:20:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc004de0 con 0x7ff5f40d15f0 2023-11-08T08:20:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f415e980 con 0x7ff5f40d15f0 2023-11-08T08:20:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.223+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc007310 con 0x7ff5f40d15f0 2023-11-08T08:20:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.224+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f406ed10 con 0x7ff5f40d15f0 2023-11-08T08:20:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.224+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f415ec60 con 0x7ff5f40d15f0 2023-11-08T08:20:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.225+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5dc01a020 con 0x7ff5f40d15f0 2023-11-08T08:20:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.225+0000 7ff5ebfff700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5d40682f0 0x7ff5d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.225+0000 7ff5f37fe700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5d40682f0 0x7ff5d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.225+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff5dc085760 con 0x7ff5f40d15f0 2023-11-08T08:20:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.226+0000 7ff5f37fe700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5d40682f0 0x7ff5d406a7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40090b0 tx=0x7ff5e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.230+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5dc052070 con 0x7ff5f40d15f0 2023-11-08T08:20:41.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.450+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5f40743f0 con 0x7ff5f40d15f0 2023-11-08T08:20:41.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.451+0000 7ff5ebfff700 1 -- 172.21.15.116:0/3414132371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7ff5dc055690 con 0x7ff5f40d15f0 2023-11-08T08:20:41.453 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:41.453 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:20:41.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5d40682f0 msgr2=0x7ff5d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5d40682f0 0x7ff5d406a7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40090b0 tx=0x7ff5e4008030 comp rx=0 tx=0).stop 2023-11-08T08:20:41.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 msgr2=0x7ff5f415cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 0x7ff5f415cbb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00c540 tx=0x7ff5dc00c850 comp rx=0 tx=0).stop 2023-11-08T08:20:41.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 shutdown_connections 2023-11-08T08:20:41.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5d40682f0 0x7ff5d406a7a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5f415e010 0x7ff5f415d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5f406e400 0x7ff5f413f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 --2- 172.21.15.116:0/3414132371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f40d15f0 0x7ff5f415cbb0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 >> 172.21.15.116:0/3414132371 conn(0x7ff5f406a270 msgr2=0x7ff5f4158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:41.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 shutdown_connections 2023-11-08T08:20:41.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.453+0000 7ff5f9ad0700 1 -- 172.21.15.116:0/3414132371 wait complete. 2023-11-08T08:20:41.457 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:20:41.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2023-11-08T08:20:41.625 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2023-11-08T08:20:41.626 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-08T08:20:41.626 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2023-11-08T08:20:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.670+0000 7f094aa9d0c0 1 Processor -- start 2023-11-08T08:20:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.670+0000 7f094aa9d0c0 1 -- start start 2023-11-08T08:20:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094aa9d0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ce3cb90 0x55f45ced2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094aa9d0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 0x55f45cec16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094aa9d0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ced4630 0x55f45cec1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094aa9d0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45ced30c0 con 0x55f45ce42e70 2023-11-08T08:20:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094aa9d0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45cec9cc0 con 0x55f45ced4630 2023-11-08T08:20:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094aa9d0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f45ced3310 con 0x55f45ce3cb90 2023-11-08T08:20:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 0x55f45cec16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 0x55f45cec16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42100/0 (socket says 172.21.15.116:42100) 2023-11-08T08:20:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 -- 172.21.15.116:0/546585632 learned_addr learned my addr 172.21.15.116:0/546585632 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f0940b89700 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ce3cb90 0x55f45ced2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094138a700 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ced4630 0x55f45cec1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 -- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ce3cb90 msgr2=0x55f45ced2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ce3cb90 0x55f45ced2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 -- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ced4630 msgr2=0x55f45cec1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ced4630 0x55f45cec1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 -- 172.21.15.116:0/546585632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0934005de0 con 0x55f45ce42e70 2023-11-08T08:20:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f094138a700 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ced4630 0x55f45cec1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093bfff700 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 0x55f45cec16a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0934000c00 tx=0x7f093400d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512f72abbf668398 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093affd700 1 -- 172.21.15.116:0/546585632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934008670 con 0x55f45ce42e70 2023-11-08T08:20:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093affd700 1 -- 172.21.15.116:0/546585632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934008b60 con 0x55f45ce42e70 2023-11-08T08:20:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.671+0000 7f093affd700 1 -- 172.21.15.116:0/546585632 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093401e450 con 0x55f45ce42e70 2023-11-08T08:20:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/546585632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 msgr2=0x55f45cec16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 0x55f45cec16a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0934000c00 tx=0x7f093400d000 comp rx=0 tx=0).stop 2023-11-08T08:20:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/546585632 shutdown_connections 2023-11-08T08:20:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ced4630 0x55f45cec1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ce3cb90 0x55f45ced2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/546585632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce42e70 0x55f45cec16a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/546585632 >> 172.21.15.116:0/546585632 conn(0x55f45ce43620 msgr2=0x55f45cdfad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/546585632 shutdown_connections 2023-11-08T08:20:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/546585632 wait complete. 2023-11-08T08:20:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.672+0000 7f094aa9d0c0 1 Processor -- start 2023-11-08T08:20:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 -- start start 2023-11-08T08:20:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce3cb90 0x55f45cec09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ce42e70 0x55f45cebfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ced4630 0x55f45cec0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45ce28cb0 con 0x55f45ce3cb90 2023-11-08T08:20:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45cecb000 con 0x55f45ce42e70 2023-11-08T08:20:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094aa9d0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f45cecb2e0 con 0x55f45ced4630 2023-11-08T08:20:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f0940b89700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce3cb90 0x55f45cec09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f0940b89700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce3cb90 0x55f45cec09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42112/0 (socket says 172.21.15.116:42112) 2023-11-08T08:20:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f0940b89700 1 -- 172.21.15.116:0/945867540 learned_addr learned my addr 172.21.15.116:0/945867540 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f094138a700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ced4630 0x55f45cec0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.673+0000 7f093bfff700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ce42e70 0x55f45cebfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0940b89700 1 -- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ced4630 msgr2=0x55f45cec0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0940b89700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f45ced4630 0x55f45cec0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0940b89700 1 -- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ce42e70 msgr2=0x55f45cebfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0940b89700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f45ce42e70 0x55f45cebfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0940b89700 1 -- 172.21.15.116:0/945867540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f45cd3e4a0 con 0x55f45ce3cb90 2023-11-08T08:20:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0940b89700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce3cb90 0x55f45cec09d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f092c009ef0 tx=0x7f092c006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0939ffb700 1 -- 172.21.15.116:0/945867540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c010040 con 0x55f45ce3cb90 2023-11-08T08:20:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0939ffb700 1 -- 172.21.15.116:0/945867540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c0092d0 con 0x55f45ce3cb90 2023-11-08T08:20:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f45cd654f0 con 0x55f45ce3cb90 2023-11-08T08:20:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f0939ffb700 1 -- 172.21.15.116:0/945867540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c015680 con 0x55f45ce3cb90 2023-11-08T08:20:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.674+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f45cd4b120 con 0x55f45ce3cb90 2023-11-08T08:20:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.675+0000 7f0939ffb700 1 -- 172.21.15.116:0/945867540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f092c0157e0 con 0x55f45ce3cb90 2023-11-08T08:20:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.675+0000 7f0939ffb700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0924066c10 0x7f09240690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f0939ffb700 1 -- 172.21.15.116:0/945867540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f092c083ef0 con 0x55f45ce3cb90 2023-11-08T08:20:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f093bfff700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0924066c10 0x7f09240690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55f45cec0f10 0x55f45cf66450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) v8 -- 0x55f45cf66990 con 0x55f45cec0f10 2023-11-08T08:20:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f093bfff700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0924066c10 0x7f09240690c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f093400a4f0 tx=0x7f0934010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f094138a700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55f45cec0f10 0x55f45cf66450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.676+0000 7f094138a700 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55f45cec0f10 0x55f45cf66450 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094138a700 1 -- 172.21.15.116:0/945867540 <== osd.7 v2:172.21.15.129:6816/121422 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f093c00d8e0 con 0x55f45cec0f10 2023-11-08T08:20:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55f45cec0f10 msgr2=0x55f45cf66450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55f45cec0f10 0x55f45cf66450 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0924066c10 msgr2=0x7f09240690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0924066c10 0x7f09240690c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f093400a4f0 tx=0x7f0934010000 comp rx=0 tx=0).stop 2023-11-08T08:20:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce3cb90 msgr2=0x55f45cec09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 --2- 172.21.15.116:0/945867540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f45ce3cb90 0x55f45cec09d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f092c009ef0 tx=0x7f092c006660 comp rx=0 tx=0).stop 2023-11-08T08:20:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094138a700 1 -- 172.21.15.116:0/945867540 reap_dead start 2023-11-08T08:20:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 shutdown_connections 2023-11-08T08:20:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 >> 172.21.15.116:0/945867540 conn(0x55f45ce43620 msgr2=0x55f45ce23640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:41.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 shutdown_connections 2023-11-08T08:20:41.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.677+0000 7f094aa9d0c0 1 -- 172.21.15.116:0/945867540 wait complete. 2023-11-08T08:20:41.700 INFO:teuthology.orchestra.run.smithi116.stdout:Yalphaparent_a 2023-11-08T08:20:41.700 DEBUG:teuthology.orchestra.run.smithi116:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-11-08T08:20:41.813 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:20:41.813 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1099511627777, 2023-11-08T08:20:41.813 INFO:teuthology.orchestra.run.smithi116.stdout: "ancestors": [ 2023-11-08T08:20:41.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:41.814 INFO:teuthology.orchestra.run.smithi116.stdout: "dirino": 1099511627776, 2023-11-08T08:20:41.814 INFO:teuthology.orchestra.run.smithi116.stdout: "dname": "alpha", 2023-11-08T08:20:41.814 INFO:teuthology.orchestra.run.smithi116.stdout: "version": 2 2023-11-08T08:20:41.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:20:41.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:20:41.815 INFO:teuthology.orchestra.run.smithi116.stdout: "dirino": 1, 2023-11-08T08:20:41.815 INFO:teuthology.orchestra.run.smithi116.stdout: "dname": "parent_a", 2023-11-08T08:20:41.815 INFO:teuthology.orchestra.run.smithi116.stdout: "version": 4 2023-11-08T08:20:41.815 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:41.816 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-11-08T08:20:41.816 INFO:teuthology.orchestra.run.smithi116.stdout: "pool": 8, 2023-11-08T08:20:41.816 INFO:teuthology.orchestra.run.smithi116.stdout: "old_pools": [] 2023-11-08T08:20:41.816 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:20:41.816 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:41.818 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2023-11-08T08:20:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.862+0000 7f0febf6e0c0 1 Processor -- start 2023-11-08T08:20:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.862+0000 7f0febf6e0c0 1 -- start start 2023-11-08T08:20:41.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0febf6e0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 0x55d08709c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0febf6e0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d086fd54f0 0x55d086ffd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0febf6e0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d0870a8f70 0x55d087133250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0febf6e0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d086f0dfe0 con 0x55d086fd1f90 2023-11-08T08:20:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0febf6e0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d08709ce20 con 0x55d086fd54f0 2023-11-08T08:20:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0febf6e0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d086ffdf90 con 0x55d0870a8f70 2023-11-08T08:20:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 0x55d08709c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 0x55d08709c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42142/0 (socket says 172.21.15.116:42142) 2023-11-08T08:20:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 -- 172.21.15.116:0/3560522251 learned_addr learned my addr 172.21.15.116:0/3560522251 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe285b700 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d0870a8f70 0x55d087133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe1859700 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d086fd54f0 0x55d086ffd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 -- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d0870a8f70 msgr2=0x55d087133250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d0870a8f70 0x55d087133250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 -- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d086fd54f0 msgr2=0x55d086ffd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d086fd54f0 0x55d086ffd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 -- 172.21.15.116:0/3560522251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd4005d20 con 0x55d086fd1f90 2023-11-08T08:20:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe205a700 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 0x55d08709c8e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800b270 tx=0x7f0fd800b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1e857f752332d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe0857700 1 -- 172.21.15.116:0/3560522251 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014070 con 0x55d086fd1f90 2023-11-08T08:20:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe0857700 1 -- 172.21.15.116:0/3560522251 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd800f3f0 con 0x55d086fd1f90 2023-11-08T08:20:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.863+0000 7f0fe0857700 1 -- 172.21.15.116:0/3560522251 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd800f760 con 0x55d086fd1f90 2023-11-08T08:20:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/3560522251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 msgr2=0x55d08709c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 0x55d08709c8e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800b270 tx=0x7f0fd800b580 comp rx=0 tx=0).stop 2023-11-08T08:20:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/3560522251 shutdown_connections 2023-11-08T08:20:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d0870a8f70 0x55d087133250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd1f90 0x55d08709c8e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/3560522251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d086fd54f0 0x55d086ffd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/3560522251 >> 172.21.15.116:0/3560522251 conn(0x55d08701a8d0 msgr2=0x55d0870a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/3560522251 shutdown_connections 2023-11-08T08:20:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/3560522251 wait complete. 2023-11-08T08:20:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 Processor -- start 2023-11-08T08:20:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.864+0000 7f0febf6e0c0 1 -- start start 2023-11-08T08:20:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0febf6e0c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d086fd1f90 0x55d0870173a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0febf6e0c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd54f0 0x55d087093ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0febf6e0c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d0870a8f70 0x55d087094530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0febf6e0c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d087144e70 con 0x55d086fd54f0 2023-11-08T08:20:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0febf6e0c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d087144260 con 0x55d0870a8f70 2023-11-08T08:20:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0febf6e0c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d087144540 con 0x55d086fd1f90 2023-11-08T08:20:41.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0fe1859700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd54f0 0x55d087093ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0fe205a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d086fd1f90 0x55d0870173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0fe1859700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd54f0 0x55d087093ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42152/0 (socket says 172.21.15.116:42152) 2023-11-08T08:20:41.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0fe1859700 1 -- 172.21.15.116:0/2323911877 learned_addr learned my addr 172.21.15.116:0/2323911877 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:41.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0fe205a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d086fd1f90 0x55d0870173a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:40248/0 (socket says 172.21.15.116:40248) 2023-11-08T08:20:41.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.865+0000 7f0fe285b700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d0870a8f70 0x55d087094530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fe1859700 1 -- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d086fd1f90 msgr2=0x55d0870173a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fe1859700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55d086fd1f90 0x55d0870173a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fe1859700 1 -- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d0870a8f70 msgr2=0x55d087094530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fe1859700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d0870a8f70 0x55d087094530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fe1859700 1 -- 172.21.15.116:0/2323911877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8001b40 con 0x55d086fd54f0 2023-11-08T08:20:41.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fe1859700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd54f0 0x55d087093ff0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4011000 tx=0x7f0fd4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fd37fe700 1 -- 172.21.15.116:0/2323911877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4008690 con 0x55d086fd54f0 2023-11-08T08:20:41.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fd37fe700 1 -- 172.21.15.116:0/2323911877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd4005310 con 0x55d086fd54f0 2023-11-08T08:20:41.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d086f26120 con 0x55d086fd54f0 2023-11-08T08:20:41.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0fd37fe700 1 -- 172.21.15.116:0/2323911877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd401d870 con 0x55d086fd54f0 2023-11-08T08:20:41.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.866+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d086f19830 con 0x55d086fd54f0 2023-11-08T08:20:41.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.867+0000 7f0fd37fe700 1 -- 172.21.15.116:0/2323911877 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0fd402d070 con 0x55d086fd54f0 2023-11-08T08:20:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.867+0000 7f0fd37fe700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fc4066c10 0x7f0fc40690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0fd37fe700 1 -- 172.21.15.116:0/2323911877 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0fd4085d80 con 0x55d086fd54f0 2023-11-08T08:20:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0fe205a700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fc4066c10 0x7f0fc40690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55d0870178e0 0x55d08713fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) v8 -- 0x55d087140530 con 0x55d0870178e0 2023-11-08T08:20:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0fe205a700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fc4066c10 0x7f0fc40690c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55d08709d440 tx=0x7f0fd8011040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0fe285b700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55d0870178e0 0x55d08713fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.868+0000 7f0fe285b700 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55d0870178e0 0x55d08713fff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0fe285b700 1 -- 172.21.15.116:0/2323911877 <== osd.7 v2:172.21.15.129:6816/121422 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v44'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f0fdc00e910 con 0x55d0870178e0 2023-11-08T08:20:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55d0870178e0 msgr2=0x55d08713fff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x55d0870178e0 0x55d08713fff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fc4066c10 msgr2=0x7f0fc40690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fc4066c10 0x7f0fc40690c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55d08709d440 tx=0x7f0fd8011040 comp rx=0 tx=0).stop 2023-11-08T08:20:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd54f0 msgr2=0x55d087093ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 --2- 172.21.15.116:0/2323911877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d086fd54f0 0x55d087093ff0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4011000 tx=0x7f0fd4002fb0 comp rx=0 tx=0).stop 2023-11-08T08:20:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0fe285b700 1 -- 172.21.15.116:0/2323911877 reap_dead start 2023-11-08T08:20:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 shutdown_connections 2023-11-08T08:20:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 >> 172.21.15.116:0/2323911877 conn(0x55d08701a8d0 msgr2=0x55d086ffd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 shutdown_connections 2023-11-08T08:20:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:41.870+0000 7f0febf6e0c0 1 -- 172.21.15.116:0/2323911877 wait complete. 2023-11-08T08:20:41.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo daemon-helper kill ceph --cluster ceph -w 2023-11-08T08:20:41.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2023-11-08T08:20:42.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.012+0000 7fcf56b9c700 1 Processor -- start 2023-11-08T08:20:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.012+0000 7fcf56b9c700 1 -- start start 2023-11-08T08:20:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf56b9c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50071320 0x7fcf500af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf56b9c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf50161160 0x7fcf5006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf56b9c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500cf3e0 con 0x7fcf5015cc50 2023-11-08T08:20:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf56b9c700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf500cd800 con 0x7fcf50161160 2023-11-08T08:20:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf56b9c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500cdae0 con 0x7fcf50071320 2023-11-08T08:20:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf55139700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf50161160 0x7fcf5006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf55139700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf50161160 0x7fcf5006a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42154/0 (socket says 172.21.15.116:42154) 2023-11-08T08:20:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.013+0000 7fcf55139700 1 -- 172.21.15.116:0/1363013369 learned_addr learned my addr 172.21.15.116:0/1363013369 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1363013369 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50071320 0x7fcf500af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf55139700 1 -- 172.21.15.116:0/1363013369 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50071320 msgr2=0x7fcf500af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf55139700 1 --2- 172.21.15.116:0/1363013369 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50071320 0x7fcf500af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf55139700 1 -- 172.21.15.116:0/1363013369 >> v1:172.21.15.129:6789/0 conn(0x7fcf5015cc50 legacy=0x7fcf50070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf55139700 1 -- 172.21.15.116:0/1363013369 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500cdf70 con 0x7fcf50161160 2023-11-08T08:20:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf55139700 1 --2- 172.21.15.116:0/1363013369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf50161160 0x7fcf5006a270 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcf400098f0 tx=0x7fcf40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a91f3239d9c73c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf4f7fe700 1 -- 172.21.15.116:0/1363013369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40011830 con 0x7fcf50161160 2023-11-08T08:20:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf4f7fe700 1 -- 172.21.15.116:0/1363013369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf40011990 con 0x7fcf50161160 2023-11-08T08:20:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.014+0000 7fcf4f7fe700 1 -- 172.21.15.116:0/1363013369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40011d20 con 0x7fcf50161160 2023-11-08T08:20:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1363013369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf50161160 msgr2=0x7fcf5006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1363013369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf50161160 0x7fcf5006a270 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcf400098f0 tx=0x7fcf40009c00 comp rx=0 tx=0).stop 2023-11-08T08:20:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1363013369 shutdown_connections 2023-11-08T08:20:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1363013369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf50161160 0x7fcf5006a270 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1363013369 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50071320 0x7fcf500af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1363013369 >> 172.21.15.116:0/1363013369 conn(0x7fcf500cfc30 msgr2=0x7fcf500d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1363013369 shutdown_connections 2023-11-08T08:20:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.015+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1363013369 wait complete. 2023-11-08T08:20:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.016+0000 7fcf56b9c700 1 Processor -- start 2023-11-08T08:20:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.016+0000 7fcf56b9c700 1 -- start start 2023-11-08T08:20:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf50071320 0x7fcf50156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 0x7fcf5006dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50161160 0x7fcf5006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5015d460 con 0x7fcf5015cc50 2023-11-08T08:20:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500b4b10 con 0x7fcf50071320 2023-11-08T08:20:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500b4270 con 0x7fcf50161160 2023-11-08T08:20:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 0x7fcf5006dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 0x7fcf5006dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42168/0 (socket says 172.21.15.116:42168) 2023-11-08T08:20:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1783079817 learned_addr learned my addr 172.21.15.116:0/1783079817 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf55139700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50161160 0x7fcf5006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf54938700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf50071320 0x7fcf50156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50161160 msgr2=0x7fcf5006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50161160 0x7fcf5006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf50071320 msgr2=0x7fcf50156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf50071320 0x7fcf50156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1783079817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf40009550 con 0x7fcf5015cc50 2023-11-08T08:20:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 0x7fcf5006dcf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800e980 tx=0x7fcf3800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4d7fa700 1 -- 172.21.15.116:0/1783079817 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38009710 con 0x7fcf5015cc50 2023-11-08T08:20:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4d7fa700 1 -- 172.21.15.116:0/1783079817 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3800c690 con 0x7fcf5015cc50 2023-11-08T08:20:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf500cf610 con 0x7fcf5015cc50 2023-11-08T08:20:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.017+0000 7fcf4d7fa700 1 -- 172.21.15.116:0/1783079817 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3800ca40 con 0x7fcf5015cc50 2023-11-08T08:20:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.018+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501616b0 con 0x7fcf5015cc50 2023-11-08T08:20:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.019+0000 7fcf4d7fa700 1 -- 172.21.15.116:0/1783079817 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf38019030 con 0x7fcf5015cc50 2023-11-08T08:20:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.020+0000 7fcf4d7fa700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf300682f0 0x7fcf3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.020+0000 7fcf54938700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf300682f0 0x7fcf3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.020+0000 7fcf4d7fa700 1 -- 172.21.15.116:0/1783079817 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcf38085af0 con 0x7fcf5015cc50 2023-11-08T08:20:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.020+0000 7fcf54938700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf300682f0 0x7fcf3006a7a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcf480012e0 tx=0x7fcf48009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.021+0000 7fcf62d3bb80 1 Processor -- start 2023-11-08T08:20:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.021+0000 7fcf62d3bb80 1 -- start start 2023-11-08T08:20:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c209a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 0x55893c2503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c2326f0 0x55893c20abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c187eb0 con 0x55893c246b50 2023-11-08T08:20:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c0f4210 con 0x55893c2326f0 2023-11-08T08:20:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893c162e80 con 0x55893c24ffe0 2023-11-08T08:20:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 0x55893c2503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 0x55893c2503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42178/0 (socket says 172.21.15.116:42178) 2023-11-08T08:20:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1344181206 learned_addr learned my addr 172.21.15.116:0/1344181206 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf54938700 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c209a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf55139700 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c2326f0 0x55893c20abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 msgr2=0x55893c209a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c209a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c2326f0 msgr2=0x55893c20abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c2326f0 0x55893c20abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 -- 172.21.15.116:0/1344181206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893bf50860 con 0x55893c246b50 2023-11-08T08:20:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.022+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 0x55893c2503b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fcf3805db70 tx=0x7fcf3805e120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818e148c2797e4e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf467fc700 1 -- 172.21.15.116:0/1344181206 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38055500 con 0x55893c246b50 2023-11-08T08:20:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf467fc700 1 -- 172.21.15.116:0/1344181206 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38055660 con 0x55893c246b50 2023-11-08T08:20:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf467fc700 1 -- 172.21.15.116:0/1344181206 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf380557d0 con 0x55893c246b50 2023-11-08T08:20:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/1344181206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 msgr2=0x55893c2503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 0x55893c2503b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fcf3805db70 tx=0x7fcf3805e120 comp rx=0 tx=0).stop 2023-11-08T08:20:42.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/1344181206 shutdown_connections 2023-11-08T08:20:42.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c2326f0 0x55893c20abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c209a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/1344181206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c246b50 0x55893c2503b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/1344181206 >> 172.21.15.116:0/1344181206 conn(0x55893c1ffdb0 msgr2=0x55893c208fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/1344181206 shutdown_connections 2023-11-08T08:20:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.023+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/1344181206 wait complete. 2023-11-08T08:20:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.024+0000 7fcf62d3bb80 1 -- start start 2023-11-08T08:20:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.024+0000 7fcf62d3bb80 1 Processor -- start 2023-11-08T08:20:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c24c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c22c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c15c1f0 con 0x55893c2326f0 2023-11-08T08:20:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893bf51460 con 0x55893c246b50 2023-11-08T08:20:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893c0f4410 con 0x55893c24ffe0 2023-11-08T08:20:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf62d3bb80 10 client.0 _ll_register_callbacks cb 0x55893c0fa4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:20:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c24c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c24c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42180/0 (socket says 172.21.15.116:42180) 2023-11-08T08:20:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 -- 172.21.15.116:0/3813939240 learned_addr learned my addr 172.21.15.116:0/3813939240 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf55139700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c22c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 -- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 msgr2=0x55893c22c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c22c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 -- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 msgr2=0x55893c22cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf54938700 1 -- 172.21.15.116:0/3813939240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501616b0 con 0x55893c2326f0 2023-11-08T08:20:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.025+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:20:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf54938700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c24c860 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcf48014ac0 tx=0x7fcf480014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec29867d83d96db server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:20:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/3813939240 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf480135c0 con 0x55893c2326f0 2023-11-08T08:20:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/3813939240 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcf48013720 con 0x55893c2326f0 2023-11-08T08:20:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55893c160100 con 0x55893c2326f0 2023-11-08T08:20:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/3813939240 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf480138e0 con 0x55893c2326f0 2023-11-08T08:20:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 10 client.5964 fetch_fsmap learned FSMap version 24 2023-11-08T08:20:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55893c03d170 con 0x55893c2326f0 2023-11-08T08:20:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/3813939240 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55893c160100 con 0x55893c2326f0 2023-11-08T08:20:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/3813939240 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fcf48013a40 con 0x55893c2326f0 2023-11-08T08:20:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 10 client.5964 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:20:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 10 client.5964 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.b 2023-11-08T08:20:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 20 client.5964 populate_metadata read hostname 'smithi116' 2023-11-08T08:20:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2498f0 0x55893c24a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 4 client.5964 mds_command: new command op to 5781 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:20:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55893bfabfc0 con 0x55893c2498f0 2023-11-08T08:20:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.026+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2498f0 0x55893c24a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.027+0000 7fcf2f7fe700 10 client.5964 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.027+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2498f0 0x55893c24a2e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe3d47ad59685793 server_cookie=ed7d571f49d3d1b3 in_seq=0 out_seq=0 2023-11-08T08:20:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.027+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/3813939240 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55893bfabfc0 con 0x55893c2498f0 2023-11-08T08:20:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.027+0000 7fcf2f7fe700 10 client.5964 handle_command_reply: tid=0 2023-11-08T08:20:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.027+0000 7fcf62d3bb80 1 client.5964 shutdown 2023-11-08T08:20:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 msgr2=0x55893c24c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c24c860 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcf48014ac0 tx=0x7fcf480014a0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 shutdown_connections 2023-11-08T08:20:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2498f0 0x55893c24a2e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c22c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c24c860 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf2f7fe700 0 client.5964 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/3813939240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 >> 172.21.15.116:0/3813939240 conn(0x55893c1ffdb0 msgr2=0x55893c204f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 shutdown_connections 2023-11-08T08:20:42.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/3813939240 wait complete. 2023-11-08T08:20:42.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.028+0000 7fcf62d3bb80 20 client.5964 trim_cache size 0 max 16384 2023-11-08T08:20:42.062 INFO:tasks.ceph.mds.b.smithi129.stderr:2023-11-08T08:20:42.051+0000 7f9ff207d700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5922=0-4194304@1} caps={5922=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5922 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55eecefe9700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-08T08:20:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.037+0000 7f45b6561700 1 Processor -- start 2023-11-08T08:20:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.037+0000 7f45b6561700 1 -- start start 2023-11-08T08:20:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b6561700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f45b015a420 0x7f45b006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b6561700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00b54a0 0x7f45b006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b6561700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b00b4e20 con 0x7f45b00716d0 2023-11-08T08:20:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b6561700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b015e1f0 con 0x7f45b015a420 2023-11-08T08:20:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b6561700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015e4d0 con 0x7f45b00b54a0 2023-11-08T08:20:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b4afe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f45b00716d0 0x7f45b00cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52134/0 (socket says 172.21.15.116:52134) 2023-11-08T08:20:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45b4afe700 1 -- 172.21.15.116:0/3292148129 learned_addr learned my addr 172.21.15.116:0/3292148129 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:20:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45af7fe700 1 --2- 172.21.15.116:0/3292148129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00b54a0 0x7f45b006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45affff700 1 --2- 172.21.15.116:0/3292148129 >> v2:172.21.15.129:3300/0 conn(0x7f45b015a420 0x7f45b006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45aeffd700 1 -- 172.21.15.116:0/3292148129 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124558377 0 0) 0x7f45b00b4e20 con 0x7f45b00716d0 2023-11-08T08:20:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45aeffd700 1 -- 172.21.15.116:0/3292148129 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45980035b0 con 0x7f45b00716d0 2023-11-08T08:20:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45af7fe700 1 -- 172.21.15.116:0/3292148129 >> v2:172.21.15.129:3300/0 conn(0x7f45b015a420 msgr2=0x7f45b006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45af7fe700 1 --2- 172.21.15.116:0/3292148129 >> v2:172.21.15.129:3300/0 conn(0x7f45b015a420 0x7f45b006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45af7fe700 1 -- 172.21.15.116:0/3292148129 >> v1:172.21.15.116:6789/0 conn(0x7f45b00716d0 legacy=0x7f45b00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45af7fe700 1 -- 172.21.15.116:0/3292148129 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b015e7b0 con 0x7f45b00b54a0 2023-11-08T08:20:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.038+0000 7f45af7fe700 1 --2- 172.21.15.116:0/3292148129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00b54a0 0x7f45b006a7b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009870 tx=0x7f45a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf104cf2121926c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45aeffd700 1 -- 172.21.15.116:0/3292148129 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4011830 con 0x7f45b00b54a0 2023-11-08T08:20:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45aeffd700 1 -- 172.21.15.116:0/3292148129 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4011990 con 0x7f45b00b54a0 2023-11-08T08:20:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45aeffd700 1 -- 172.21.15.116:0/3292148129 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4011d20 con 0x7f45b00b54a0 2023-11-08T08:20:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 -- 172.21.15.116:0/3292148129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00b54a0 msgr2=0x7f45b006a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 --2- 172.21.15.116:0/3292148129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00b54a0 0x7f45b006a7b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009870 tx=0x7f45a4009b80 comp rx=0 tx=0).stop 2023-11-08T08:20:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 -- 172.21.15.116:0/3292148129 shutdown_connections 2023-11-08T08:20:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 --2- 172.21.15.116:0/3292148129 >> v2:172.21.15.129:3300/0 conn(0x7f45b015a420 0x7f45b006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 --2- 172.21.15.116:0/3292148129 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00b54a0 0x7f45b006a7b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 -- 172.21.15.116:0/3292148129 >> 172.21.15.116:0/3292148129 conn(0x7f45b00cc660 msgr2=0x7f45b00cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 -- 172.21.15.116:0/3292148129 shutdown_connections 2023-11-08T08:20:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.039+0000 7f45b6561700 1 -- 172.21.15.116:0/3292148129 wait complete. 2023-11-08T08:20:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.040+0000 7f45b6561700 1 Processor -- start 2023-11-08T08:20:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.040+0000 7f45b6561700 1 -- start start 2023-11-08T08:20:42.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b6561700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00716d0 0x7f45b015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b6561700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b00b54a0 0x7f45b015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b6561700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45b015a420 0x7f45b014d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b6561700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015ca30 con 0x7f45b015a420 2023-11-08T08:20:42.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b6561700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015d390 con 0x7f45b00b54a0 2023-11-08T08:20:42.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b6561700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015b740 con 0x7f45b00716d0 2023-11-08T08:20:42.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b4afe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45b015a420 0x7f45b014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b4afe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45b015a420 0x7f45b014d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42182/0 (socket says 172.21.15.116:42182) 2023-11-08T08:20:42.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45b4afe700 1 -- 172.21.15.116:0/1512686460 learned_addr learned my addr 172.21.15.116:0/1512686460 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45af7fe700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b00b54a0 0x7f45b015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.041+0000 7f45affff700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00716d0 0x7f45b015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b4afe700 1 -- 172.21.15.116:0/1512686460 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00716d0 msgr2=0x7f45b015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b4afe700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f45b00716d0 0x7f45b015ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b4afe700 1 -- 172.21.15.116:0/1512686460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b00b54a0 msgr2=0x7f45b015b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b4afe700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b00b54a0 0x7f45b015b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b4afe700 1 -- 172.21.15.116:0/1512686460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40094a0 con 0x7f45b015a420 2023-11-08T08:20:42.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b4afe700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45b015a420 0x7f45b014d830 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f45a800cad0 tx=0x7f45a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a800a940 con 0x7f45b015a420 2023-11-08T08:20:42.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a800aaa0 con 0x7f45b015a420 2023-11-08T08:20:42.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a800ae30 con 0x7f45b015a420 2023-11-08T08:20:42.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b6561700 1 -- 172.21.15.116:0/1512686460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b015ed70 con 0x7f45b015a420 2023-11-08T08:20:42.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.042+0000 7f45b6561700 1 -- 172.21.15.116:0/1512686460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b015ea90 con 0x7f45b015a420 2023-11-08T08:20:42.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.043+0000 7f45b6561700 1 -- 172.21.15.116:0/1512686460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f45b015de80 con 0x7f45b015a420 2023-11-08T08:20:42.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.043+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f45a8019030 con 0x7f45b015a420 2023-11-08T08:20:42.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.044+0000 7f45acff9700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f459c0682f0 0x7f459c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.044+0000 7f45affff700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f459c0682f0 0x7f459c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.044+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f45a8085df0 con 0x7f45b015a420 2023-11-08T08:20:42.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.044+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+502 (secure 0 0 0) 0x7f45a8086270 con 0x7f45b015a420 2023-11-08T08:20:42.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.044+0000 7f45affff700 1 --2- 172.21.15.116:0/1512686460 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f459c0682f0 0x7f459c06a7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009be0 tx=0x7f45a0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.044+0000 7f45b6561700 1 -- 172.21.15.116:0/1512686460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f45b015cb70 con 0x7f45b015a420 2023-11-08T08:20:42.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.045+0000 7fcf62d3bb80 1 Processor -- start 2023-11-08T08:20:42.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.045+0000 7fcf62d3bb80 1 -- start start 2023-11-08T08:20:42.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c2326f0 0x55893c24c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 0x55893c268760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c22fb00 con 0x55893c24ffe0 2023-11-08T08:20:42.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c208830 con 0x55893c246b50 2023-11-08T08:20:42.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893c22fe80 con 0x55893c2326f0 2023-11-08T08:20:42.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 0x55893c268760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf4ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 0x55893c268760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42184/0 (socket says 172.21.15.116:42184) 2023-11-08T08:20:42.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf54938700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c2326f0 0x55893c24c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 -- 172.21.15.116:0/12163765 learned_addr learned my addr 172.21.15.116:0/12163765 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 -- 172.21.15.116:0/12163765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c2326f0 msgr2=0x55893c24c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c2326f0 0x55893c24c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 -- 172.21.15.116:0/12163765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 msgr2=0x55893c22e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 -- 172.21.15.116:0/12163765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893c238ef0 con 0x55893c24ffe0 2023-11-08T08:20:42.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.046+0000 7fcf55139700 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 0x55893c268760 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcf40014040 tx=0x7fcf40010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97bacffe749cd73 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf46ffd700 1 -- 172.21.15.116:0/12163765 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4001f040 con 0x55893c24ffe0 2023-11-08T08:20:42.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf46ffd700 1 -- 172.21.15.116:0/12163765 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf400117f0 con 0x55893c24ffe0 2023-11-08T08:20:42.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf46ffd700 1 -- 172.21.15.116:0/12163765 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40011d20 con 0x55893c24ffe0 2023-11-08T08:20:42.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/12163765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 msgr2=0x55893c268760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 0x55893c268760 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcf40014040 tx=0x7fcf40010000 comp rx=0 tx=0).stop 2023-11-08T08:20:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/12163765 shutdown_connections 2023-11-08T08:20:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c24ffe0 0x55893c268760 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c2326f0 0x55893c24c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/12163765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c22e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/12163765 >> 172.21.15.116:0/12163765 conn(0x55893c1ffdb0 msgr2=0x55893c207310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/12163765 shutdown_connections 2023-11-08T08:20:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.047+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/12163765 wait complete. 2023-11-08T08:20:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.048+0000 7fcf62d3bb80 1 -- start start 2023-11-08T08:20:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.048+0000 7fcf62d3bb80 1 Processor -- start 2023-11-08T08:20:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c2b9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c2b9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c230050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c24b530 con 0x55893c2326f0 2023-11-08T08:20:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c2ba390 con 0x55893c246b50 2023-11-08T08:20:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893c26cee0 con 0x55893c24ffe0 2023-11-08T08:20:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf62d3bb80 10 client.0 _ll_register_callbacks cb 0x55893c0fa4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:20:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c2b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c2b9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42200/0 (socket says 172.21.15.116:42200) 2023-11-08T08:20:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 -- 172.21.15.116:0/82243774 learned_addr learned my addr 172.21.15.116:0/82243774 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c2b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf55139700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c230050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 -- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 msgr2=0x55893c230050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c230050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 -- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 msgr2=0x55893c2b9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c2b9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 -- 172.21.15.116:0/82243774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf500cdf70 con 0x55893c2326f0 2023-11-08T08:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf55139700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c230050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf54938700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c2b9400 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800fc20 tx=0x7fcf48015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705d13df9c14d89b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.049+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/82243774 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48038040 con 0x55893c2326f0 2023-11-08T08:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/82243774 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcf480139d0 con 0x55893c2326f0 2023-11-08T08:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf38055660 con 0x55893c2326f0 2023-11-08T08:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/82243774 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf48013b90 con 0x55893c2326f0 2023-11-08T08:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 10 client.5976 fetch_fsmap learned FSMap version 24 2023-11-08T08:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55893c109be0 con 0x55893c2326f0 2023-11-08T08:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/82243774 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38055660 con 0x55893c2326f0 2023-11-08T08:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/82243774 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fcf4803d430 con 0x55893c2326f0 2023-11-08T08:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 10 client.5976 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 10 client.5976 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.b 2023-11-08T08:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 20 client.5976 populate_metadata read hostname 'smithi116' 2023-11-08T08:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2acab0 0x55893c230590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 4 client.5976 mds_command: new command op to 5781 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2023-11-08T08:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x55893bfe1110 con 0x55893c2acab0 2023-11-08T08:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2acab0 0x55893c230590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf2f7fe700 10 client.5976 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.050+0000 7fcf4ffff700 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2acab0 0x55893c230590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5d8689c0e197cef server_cookie=9174acc64025e5a0 in_seq=0 out_seq=0 2023-11-08T08:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf2f7fe700 1 -- 172.21.15.116:0/82243774 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55893bfe1110 con 0x55893c2acab0 2023-11-08T08:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf2f7fe700 10 client.5976 handle_command_reply: tid=0 2023-11-08T08:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 client.5976 shutdown 2023-11-08T08:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 msgr2=0x55893c2b9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c2b9400 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800fc20 tx=0x7fcf48015000 comp rx=0 tx=0).stop 2023-11-08T08:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 shutdown_connections 2023-11-08T08:20:42.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x55893c2acab0 0x55893c230590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55893c24ffe0 0x55893c230050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55893c2326f0 0x55893c2b9400 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf2f7fe700 0 client.5976 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 --2- 172.21.15.116:0/82243774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55893c246b50 0x55893c2b9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 >> 172.21.15.116:0/82243774 conn(0x55893c1ffdb0 msgr2=0x55893c207310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 shutdown_connections 2023-11-08T08:20:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.051+0000 7fcf62d3bb80 1 -- 172.21.15.116:0/82243774 wait complete. 2023-11-08T08:20:42.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf62d3bb80 20 client.5976 trim_cache size 0 max 16384 2023-11-08T08:20:42.104 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:20:42.104 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2023-11-08T08:20:42.104 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "9b5b4a63-54c0-4e59-b190-08b4a659a53a", 2023-11-08T08:20:42.105 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2023-11-08T08:20:42.105 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:20:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf300682f0 msgr2=0x7fcf3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf300682f0 0x7fcf3006a7a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcf480012e0 tx=0x7fcf48009040 comp rx=0 tx=0).stop 2023-11-08T08:20:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 msgr2=0x7fcf5006dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 0x7fcf5006dcf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800e980 tx=0x7fcf3800ec90 comp rx=0 tx=0).stop 2023-11-08T08:20:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 shutdown_connections 2023-11-08T08:20:42.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf50161160 0x7fcf5006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf300682f0 0x7fcf3006a7a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf50071320 0x7fcf50156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 --2- 172.21.15.116:0/1783079817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf5015cc50 0x7fcf5006dcf0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 >> 172.21.15.116:0/1783079817 conn(0x7fcf500cfc30 msgr2=0x7fcf50157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 shutdown_connections 2023-11-08T08:20:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.052+0000 7fcf56b9c700 1 -- 172.21.15.116:0/1783079817 wait complete. 2023-11-08T08:20:42.109 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2023-11-08T08:20:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.231+0000 7fe90d1a1700 1 Processor -- start 2023-11-08T08:20:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.231+0000 7fe90d1a1700 1 -- start start 2023-11-08T08:20:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90d1a1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe908159290 0x7fe9080d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90d1a1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe90806eea0 0x7fe908070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90d1a1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90806f340 con 0x7fe908159660 2023-11-08T08:20:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90d1a1700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90806d260 con 0x7fe90806eea0 2023-11-08T08:20:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90d1a1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9080cb3e0 con 0x7fe908159290 2023-11-08T08:20:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90c9a0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe90806eea0 0x7fe908070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90c9a0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe90806eea0 0x7fe908070a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42210/0 (socket says 172.21.15.116:42210) 2023-11-08T08:20:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.232+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2992439336 learned_addr learned my addr 172.21.15.116:0/2992439336 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe907fff700 1 --2- 172.21.15.116:0/2992439336 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe908159290 0x7fe9080d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2992439336 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe908159290 msgr2=0x7fe9080d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/2992439336 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe908159290 0x7fe9080d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2992439336 >> v1:172.21.15.129:6789/0 conn(0x7fe908159660 legacy=0x7fe90806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2992439336 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9080ca7d0 con 0x7fe90806eea0 2023-11-08T08:20:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/2992439336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe90806eea0 0x7fe908070a00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe900009e50 tx=0x7fe90000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9905d1ba360377eb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe906ffd700 1 -- 172.21.15.116:0/2992439336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90000e070 con 0x7fe90806eea0 2023-11-08T08:20:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe906ffd700 1 -- 172.21.15.116:0/2992439336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90000b9c0 con 0x7fe90806eea0 2023-11-08T08:20:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe906ffd700 1 -- 172.21.15.116:0/2992439336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90000bd70 con 0x7fe90806eea0 2023-11-08T08:20:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2992439336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe90806eea0 msgr2=0x7fe908070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.233+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2992439336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe90806eea0 0x7fe908070a00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe900009e50 tx=0x7fe90000b330 comp rx=0 tx=0).stop 2023-11-08T08:20:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.234+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2992439336 shutdown_connections 2023-11-08T08:20:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.234+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2992439336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe90806eea0 0x7fe908070a00 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.234+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2992439336 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe908159290 0x7fe9080d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.234+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2992439336 >> 172.21.15.116:0/2992439336 conn(0x7fe9080b2310 msgr2=0x7fe9080cfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.234+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2992439336 shutdown_connections 2023-11-08T08:20:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.234+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2992439336 wait complete. 2023-11-08T08:20:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.235+0000 7fe90d1a1700 1 Processor -- start 2023-11-08T08:20:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.235+0000 7fe90d1a1700 1 -- start start 2023-11-08T08:20:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe90806eea0 0x7fe9080ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe908159290 0x7fe90806a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 0x7fe90806a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9080aea60 con 0x7fe908159660 2023-11-08T08:20:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9081f30c0 con 0x7fe908159290 2023-11-08T08:20:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9081f6650 con 0x7fe90806eea0 2023-11-08T08:20:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 0x7fe90806a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 0x7fe90806a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42224/0 (socket says 172.21.15.116:42224) 2023-11-08T08:20:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2777322272 learned_addr learned my addr 172.21.15.116:0/2777322272 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe907fff700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe90806eea0 0x7fe9080ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe9077fe700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe908159290 0x7fe90806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe90806eea0 msgr2=0x7fe9080ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe90806eea0 0x7fe9080ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe908159290 msgr2=0x7fe90806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe908159290 0x7fe90806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 -- 172.21.15.116:0/2777322272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900009b30 con 0x7fe908159660 2023-11-08T08:20:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 0x7fe90806a7b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe9080cd610 tx=0x7fe900004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe904ff9700 1 -- 172.21.15.116:0/2777322272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900004800 con 0x7fe908159660 2023-11-08T08:20:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe904ff9700 1 -- 172.21.15.116:0/2777322272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9000092e0 con 0x7fe908159660 2023-11-08T08:20:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9080cad90 con 0x7fe908159660 2023-11-08T08:20:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe904ff9700 1 -- 172.21.15.116:0/2777322272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90001c680 con 0x7fe908159660 2023-11-08T08:20:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.236+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9080caab0 con 0x7fe908159660 2023-11-08T08:20:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.238+0000 7fe904ff9700 1 -- 172.21.15.116:0/2777322272 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe900022030 con 0x7fe908159660 2023-11-08T08:20:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.238+0000 7fe904ff9700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe8f40682f0 0x7fe8f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.238+0000 7fe904ff9700 1 -- 172.21.15.116:0/2777322272 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe9000873e0 con 0x7fe908159660 2023-11-08T08:20:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.238+0000 7fe907fff700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe8f40682f0 0x7fe8f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.238+0000 7fe907fff700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe8f40682f0 0x7fe8f406a7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006b20 tx=0x7fe8fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.240+0000 7fe91a5a2b80 1 Processor -- start 2023-11-08T08:20:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.240+0000 7fe91a5a2b80 1 -- start start 2023-11-08T08:20:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eead8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eeaadd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaae270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641eea2ae30 con 0x5641eead8280 2023-11-08T08:20:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641ee997190 con 0x5641eeaaccf0 2023-11-08T08:20:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641eea05e00 con 0x5641eeae9ca0 2023-11-08T08:20:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eead8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eead8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42228/0 (socket says 172.21.15.116:42228) 2023-11-08T08:20:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 -- 172.21.15.116:0/1284898359 learned_addr learned my addr 172.21.15.116:0/1284898359 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe9077fe700 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eeaadd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 -- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 msgr2=0x5641eeaae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 -- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 msgr2=0x5641eeaadd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eeaadd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 -- 172.21.15.116:0/1284898359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641ee7f37e0 con 0x5641eead8280 2023-11-08T08:20:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe907fff700 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eead8650 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc011950 tx=0x7fe8fc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9f59f3e71b3de3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe8edffb700 1 -- 172.21.15.116:0/1284898359 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc015740 con 0x5641eead8280 2023-11-08T08:20:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe8edffb700 1 -- 172.21.15.116:0/1284898359 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc0158a0 con 0x5641eead8280 2023-11-08T08:20:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.241+0000 7fe8edffb700 1 -- 172.21.15.116:0/1284898359 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc015c10 con 0x5641eead8280 2023-11-08T08:20:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/1284898359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 msgr2=0x5641eead8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eead8650 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc011950 tx=0x7fe8fc014000 comp rx=0 tx=0).stop 2023-11-08T08:20:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/1284898359 shutdown_connections 2023-11-08T08:20:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eead8650 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/1284898359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eeaadd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/1284898359 >> 172.21.15.116:0/1284898359 conn(0x5641eeaa2d30 msgr2=0x5641eeaac0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/1284898359 shutdown_connections 2023-11-08T08:20:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/1284898359 wait complete. 2023-11-08T08:20:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.242+0000 7fe91a5a2b80 1 -- start start 2023-11-08T08:20:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 Processor -- start 2023-11-08T08:20:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeaf5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eead8280 0x5641eead0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eead7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641ee9ff170 con 0x5641eeaaccf0 2023-11-08T08:20:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641ee7f43e0 con 0x5641eead8280 2023-11-08T08:20:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641ee997390 con 0x5641eeae9ca0 2023-11-08T08:20:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.243+0000 7fe91a5a2b80 10 client.0 _ll_register_callbacks cb 0x5641ee99d450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:20:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeaf5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeaf5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50146/0 (socket says 172.21.15.116:50146) 2023-11-08T08:20:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 -- 172.21.15.116:0/586796709 learned_addr learned my addr 172.21.15.116:0/586796709 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe9077fe700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eead8280 0x5641eead0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eead7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 -- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 msgr2=0x5641eead7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eead7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 -- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eead8280 msgr2=0x5641eead0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eead8280 0x5641eead0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 -- 172.21.15.116:0/586796709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc00e040 con 0x5641eeaaccf0 2023-11-08T08:20:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe907fff700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeaf5df0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc023040 tx=0x7fe8fc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8817b5f97a50af server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe8e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:20:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/586796709 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc015e20 con 0x5641eeaaccf0 2023-11-08T08:20:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/586796709 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe8fc036470 con 0x5641eeaaccf0 2023-11-08T08:20:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641eea03080 con 0x5641eeaaccf0 2023-11-08T08:20:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/586796709 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8fc036670 con 0x5641eeaaccf0 2023-11-08T08:20:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe91a5a2b80 10 client.5988 fetch_fsmap learned FSMap version 24 2023-11-08T08:20:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x5641ee8e00f0 con 0x5641eeaaccf0 2023-11-08T08:20:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.244+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/586796709 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5641eea03080 con 0x5641eeaaccf0 2023-11-08T08:20:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/586796709 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe8fc0367d0 con 0x5641eeaaccf0 2023-11-08T08:20:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe91a5a2b80 10 client.5988 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:20:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe91a5a2b80 10 client.5988 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.b 2023-11-08T08:20:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe91a5a2b80 20 client.5988 populate_metadata read hostname 'smithi116' 2023-11-08T08:20:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaf3c20 0x5641eead26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe91a5a2b80 4 client.5988 mds_command: new command op to 5781 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:20:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641ee84ef40 con 0x5641eeaf3c20 2023-11-08T08:20:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe9077fe700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaf3c20 0x5641eead26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe8e6ffd700 10 client.5988 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.245+0000 7fe9077fe700 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaf3c20 0x5641eead26e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=979d2105bb5404cc server_cookie=224c6a0b845b304e in_seq=0 out_seq=0 2023-11-08T08:20:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/586796709 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5641ee84ef40 con 0x5641eeaf3c20 2023-11-08T08:20:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe8e6ffd700 10 client.5988 handle_command_reply: tid=0 2023-11-08T08:20:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 client.5988 shutdown 2023-11-08T08:20:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 msgr2=0x5641eeaf5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeaf5df0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc023040 tx=0x7fe8fc0047c0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 shutdown_connections 2023-11-08T08:20:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaf3c20 0x5641eead26e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eead7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeaf5df0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe8e6ffd700 0 client.5988 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/586796709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eead8280 0x5641eead0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 >> 172.21.15.116:0/586796709 conn(0x5641eeaa2d30 msgr2=0x5641eeaad110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 shutdown_connections 2023-11-08T08:20:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/586796709 wait complete. 2023-11-08T08:20:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.246+0000 7fe91a5a2b80 20 client.5988 trim_cache size 0 max 16384 2023-11-08T08:20:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 Processor -- start 2023-11-08T08:20:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 -- start start 2023-11-08T08:20:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eead5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eeaf6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaf6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641eeacdf80 con 0x5641eead8280 2023-11-08T08:20:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641eeb059e0 con 0x5641eeaaccf0 2023-11-08T08:20:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641eead8ea0 con 0x5641eeae9ca0 2023-11-08T08:20:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe9077fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eeaf6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe9077fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eeaf6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50152/0 (socket says 172.21.15.116:50152) 2023-11-08T08:20:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe9077fe700 1 -- 172.21.15.116:0/3315845679 learned_addr learned my addr 172.21.15.116:0/3315845679 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaf6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.263+0000 7fe907fff700 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eead5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe9077fe700 1 -- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 msgr2=0x5641eeaf6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe9077fe700 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaf6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe9077fe700 1 -- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 msgr2=0x5641eead5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe9077fe700 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eead5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe9077fe700 1 -- 172.21.15.116:0/3315845679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641ee9c6330 con 0x5641eead8280 2023-11-08T08:20:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe907fff700 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eead5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:20:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe9077fe700 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eeaf6610 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe9080ac900 tx=0x7fe8f8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3a24ecc7b66261 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe8ee7fc700 1 -- 172.21.15.116:0/3315845679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f801b070 con 0x5641eead8280 2023-11-08T08:20:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe8ee7fc700 1 -- 172.21.15.116:0/3315845679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f801c040 con 0x5641eead8280 2023-11-08T08:20:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe8ee7fc700 1 -- 172.21.15.116:0/3315845679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f801d400 con 0x5641eead8280 2023-11-08T08:20:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/3315845679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 msgr2=0x5641eeaf6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eeaf6610 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe9080ac900 tx=0x7fe8f8001080 comp rx=0 tx=0).stop 2023-11-08T08:20:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/3315845679 shutdown_connections 2023-11-08T08:20:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eeae9ca0 0x5641eeaf6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeaaccf0 0x5641eead5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/3315845679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eead8280 0x5641eeaf6610 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/3315845679 >> 172.21.15.116:0/3315845679 conn(0x5641eeaa2d30 msgr2=0x5641eeaae240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/3315845679 shutdown_connections 2023-11-08T08:20:42.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.264+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/3315845679 wait complete. 2023-11-08T08:20:42.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.265+0000 7fe91a5a2b80 1 -- start start 2023-11-08T08:20:42.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.265+0000 7fe91a5a2b80 1 Processor -- start 2023-11-08T08:20:42.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeb58670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eead8280 0x5641eead3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeae9ca0 0x5641eead3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641eead4460 con 0x5641eeaaccf0 2023-11-08T08:20:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641eeac8eb0 con 0x5641eeae9ca0 2023-11-08T08:20:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641eeacc270 con 0x5641eead8280 2023-11-08T08:20:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe91a5a2b80 10 client.0 _ll_register_callbacks cb 0x5641ee99d450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:20:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeb58670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeb58670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50154/0 (socket says 172.21.15.116:50154) 2023-11-08T08:20:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 -- 172.21.15.116:0/2354011004 learned_addr learned my addr 172.21.15.116:0/2354011004 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:20:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe9077fe700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eead8280 0x5641eead3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe90c9a0700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeae9ca0 0x5641eead3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 -- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eead8280 msgr2=0x5641eead3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eead8280 0x5641eead3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 -- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeae9ca0 msgr2=0x5641eead3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:20:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeae9ca0 0x5641eead3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 -- 172.21.15.116:0/2354011004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f8001710 con 0x5641eeaaccf0 2023-11-08T08:20:42.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe907fff700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeb58670 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc015870 tx=0x7fe8fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bb97de3cf17b35 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:20:42.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe8e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:20:42.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.266+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/2354011004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc036ea0 con 0x5641eeaaccf0 2023-11-08T08:20:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/2354011004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe8fc036c90 con 0x5641eeaaccf0 2023-11-08T08:20:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8fc0158a0 con 0x5641eeaaccf0 2023-11-08T08:20:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/2354011004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8fc006b50 con 0x5641eeaaccf0 2023-11-08T08:20:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 10 client.5994 fetch_fsmap learned FSMap version 24 2023-11-08T08:20:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x5641ee854220 con 0x5641eeaaccf0 2023-11-08T08:20:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/2354011004 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc0158a0 con 0x5641eeaaccf0 2023-11-08T08:20:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/2354011004 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe8fc021790 con 0x5641eeaaccf0 2023-11-08T08:20:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 10 client.5994 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:20:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 10 client.5994 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.b 2023-11-08T08:20:42.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 20 client.5994 populate_metadata read hostname 'smithi116' 2023-11-08T08:20:42.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaadba0 0x5641eeaef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:20:42.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 4 client.5994 mds_command: new command op to 5781 tid=0[{"prefix": "scrub status"}] 2023-11-08T08:20:42.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5641ee884090 con 0x5641eeaadba0 2023-11-08T08:20:42.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe9077fe700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaadba0 0x5641eeaef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:20:42.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe8e6ffd700 10 client.5994 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.267+0000 7fe9077fe700 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaadba0 0x5641eeaef3d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9776f48327845f6c server_cookie=42e051b01ea5f812 in_seq=0 out_seq=0 2023-11-08T08:20:42.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe8e6ffd700 1 -- 172.21.15.116:0/2354011004 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x5641ee884090 con 0x5641eeaadba0 2023-11-08T08:20:42.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe8e6ffd700 10 client.5994 handle_command_reply: tid=0 2023-11-08T08:20:42.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 client.5994 shutdown 2023-11-08T08:20:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 msgr2=0x5641eeb58670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeb58670 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc015870 tx=0x7fe8fc017000 comp rx=0 tx=0).stop 2023-11-08T08:20:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 shutdown_connections 2023-11-08T08:20:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641eeae9ca0 0x5641eead3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5641eeaaccf0 0x5641eeb58670 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x5641eeaadba0 0x5641eeaef3d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 --2- 172.21.15.116:0/2354011004 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5641eead8280 0x5641eead3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe8e6ffd700 0 client.5994 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:20:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 >> 172.21.15.116:0/2354011004 conn(0x5641eeaa2d30 msgr2=0x5641eeac7810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 shutdown_connections 2023-11-08T08:20:42.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 1 -- 172.21.15.116:0/2354011004 wait complete. 2023-11-08T08:20:42.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe91a5a2b80 20 client.5994 trim_cache size 0 max 16384 2023-11-08T08:20:42.302 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:20:42.302 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-08T08:20:42.302 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2023-11-08T08:20:42.302 INFO:teuthology.orchestra.run.smithi116.stdout: "9b5b4a63-54c0-4e59-b190-08b4a659a53a": { 2023-11-08T08:20:42.302 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2023-11-08T08:20:42.303 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "9b5b4a63-54c0-4e59-b190-08b4a659a53a", 2023-11-08T08:20:42.303 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,repair" 2023-11-08T08:20:42.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:42.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:20:42.304 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:20:42.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe8f40682f0 msgr2=0x7fe8f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe8f40682f0 0x7fe8f406a7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006b20 tx=0x7fe8fc008040 comp rx=0 tx=0).stop 2023-11-08T08:20:42.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 msgr2=0x7fe90806a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:20:42.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 0x7fe90806a7b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe9080cd610 tx=0x7fe900004ad0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 shutdown_connections 2023-11-08T08:20:42.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe908159660 0x7fe90806a7b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe90806eea0 0x7fe9080ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe8f40682f0 0x7fe8f406a7a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 --2- 172.21.15.116:0/2777322272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe908159290 0x7fe90806a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:20:42.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.268+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 >> 172.21.15.116:0/2777322272 conn(0x7fe9080b2310 msgr2=0x7fe9080c4930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:20:42.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.269+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 shutdown_connections 2023-11-08T08:20:42.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:42.269+0000 7fe90d1a1700 1 -- 172.21.15.116:0/2777322272 wait complete. 2023-11-08T08:20:42.308 INFO:tasks.cephfs.filesystem:scrub status for tag:9b5b4a63-54c0-4e59-b190-08b4a659a53a - {'path': '/', 'tag': '9b5b4a63-54c0-4e59-b190-08b4a659a53a', 'options': 'recursive,repair'} 2023-11-08T08:20:43.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:43.049+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 7 ==== log(6 entries from seq 1 at 2023-11-08T08:20:42.052379+0000) v1 ==== 1078+0+0 (secure 0 0 0) 0x7f45a805e3a0 con 0x7f45b015a420 2023-11-08T08:20:43.051 INFO:teuthology.orchestra.run.smithi116.stdout: cluster: 2023-11-08T08:20:43.051 INFO:teuthology.orchestra.run.smithi116.stdout: id: 2f470591-d661-4b50-b7a5-062dfcf46b1d 2023-11-08T08:20:43.051 INFO:teuthology.orchestra.run.smithi116.stdout: health: HEALTH_OK 2023-11-08T08:20:43.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:43.052 INFO:teuthology.orchestra.run.smithi116.stdout: services: 2023-11-08T08:20:43.052 INFO:teuthology.orchestra.run.smithi116.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2023-11-08T08:20:43.052 INFO:teuthology.orchestra.run.smithi116.stdout: mgr: y(active, since 2m), standbys: x 2023-11-08T08:20:43.052 INFO:teuthology.orchestra.run.smithi116.stdout: mds: 1/1 daemons up, 3 standby 2023-11-08T08:20:43.053 INFO:teuthology.orchestra.run.smithi116.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2023-11-08T08:20:43.053 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:43.053 INFO:teuthology.orchestra.run.smithi116.stdout: data: 2023-11-08T08:20:43.053 INFO:teuthology.orchestra.run.smithi116.stdout: volumes: 1/1 healthy 2023-11-08T08:20:43.053 INFO:teuthology.orchestra.run.smithi116.stdout: pools: 4 pools, 105 pgs 2023-11-08T08:20:43.054 INFO:teuthology.orchestra.run.smithi116.stdout: objects: 23 objects, 579 KiB 2023-11-08T08:20:43.054 INFO:teuthology.orchestra.run.smithi116.stdout: usage: 72 MiB used, 720 GiB / 720 GiB avail 2023-11-08T08:20:43.054 INFO:teuthology.orchestra.run.smithi116.stdout: pgs: 105 active+clean 2023-11-08T08:20:43.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:43.054 INFO:teuthology.orchestra.run.smithi116.stdout: io: 2023-11-08T08:20:43.055 INFO:teuthology.orchestra.run.smithi116.stdout: client: 1.8 KiB/s wr, 0 op/s rd, 5 op/s wr 2023-11-08T08:20:43.055 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:43.055 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:20:43.055 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.052379+0000 mds.b [INF] scrub queued for path: / 2023-11-08T08:20:43.055 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.052384+0000 mds.b [INF] scrub summary: idle+waiting paths [/] 2023-11-08T08:20:43.056 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.052388+0000 mds.b [INF] scrub summary: active paths [/] 2023-11-08T08:20:43.056 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.053079+0000 mds.b [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2023-11-08T08:20:43.056 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.053088+0000 mds.b [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2023-11-08T08:20:43.056 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.053564+0000 mds.b [INF] scrub summary: idle+waiting paths [/] 2023-11-08T08:20:44.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:20:44.051+0000 7f45acff9700 1 -- 172.21.15.116:0/1512686460 <== mon.0 v2:172.21.15.116:3300/0 8 ==== log(1 entries from seq 7 at 2023-11-08T08:20:42.689636+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7f45a8086230 con 0x7f45b015a420 2023-11-08T08:20:44.052 INFO:teuthology.orchestra.run.smithi116.stdout:2023-11-08T08:20:42.689636+0000 mds.b [INF] scrub summary: idle 2023-11-08T08:20:49.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:20:49.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:21:12.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2023-11-08T08:21:12.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.446+0000 7f8226bd6700 1 Processor -- start 2023-11-08T08:21:12.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.446+0000 7f8226bd6700 1 -- start start 2023-11-08T08:21:12.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8226bd6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8220158d50 0x7f82200cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8226bd6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f822006d4e0 0x7f82200ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8226bd6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82200cdf00 con 0x7f8220158980 2023-11-08T08:21:12.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8226bd6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822006d170 con 0x7f8220158d50 2023-11-08T08:21:12.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8226bd6700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82200c9110 con 0x7f822006d4e0 2023-11-08T08:21:12.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8220158d50 0x7f82200cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8224972700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8220158980 0x7f82200ce390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53194/0 (socket says 172.21.15.116:53194) 2023-11-08T08:21:12.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8224972700 1 -- 172.21.15.116:0/337898875 learned_addr learned my addr 172.21.15.116:0/337898875 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:12.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8225173700 1 --2- 172.21.15.116:0/337898875 >> v2:172.21.15.129:3300/0 conn(0x7f822006d4e0 0x7f82200ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821ffff700 1 -- 172.21.15.116:0/337898875 >> v2:172.21.15.129:3300/0 conn(0x7f822006d4e0 msgr2=0x7f82200ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821ffff700 1 --2- 172.21.15.116:0/337898875 >> v2:172.21.15.129:3300/0 conn(0x7f822006d4e0 0x7f82200ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821f7fe700 1 -- 172.21.15.116:0/337898875 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901927654 0 0) 0x7f82200cdf00 con 0x7f8220158980 2023-11-08T08:21:12.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821ffff700 1 -- 172.21.15.116:0/337898875 >> v1:172.21.15.116:6789/0 conn(0x7f8220158980 legacy=0x7f82200ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821ffff700 1 -- 172.21.15.116:0/337898875 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82200c93f0 con 0x7f8220158d50 2023-11-08T08:21:12.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f8225173700 1 --2- 172.21.15.116:0/337898875 >> v2:172.21.15.129:3300/0 conn(0x7f822006d4e0 0x7f82200ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:12.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821ffff700 1 --2- 172.21.15.116:0/337898875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158d50 0x7f82200cc690 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f82200cd110 tx=0x7f820c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fce9771c971e6b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821f7fe700 1 -- 172.21.15.116:0/337898875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c011830 con 0x7f8220158d50 2023-11-08T08:21:12.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821f7fe700 1 -- 172.21.15.116:0/337898875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c011990 con 0x7f8220158d50 2023-11-08T08:21:12.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.447+0000 7f821f7fe700 1 -- 172.21.15.116:0/337898875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c011d20 con 0x7f8220158d50 2023-11-08T08:21:12.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 -- 172.21.15.116:0/337898875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158d50 msgr2=0x7f82200cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 --2- 172.21.15.116:0/337898875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158d50 0x7f82200cc690 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f82200cd110 tx=0x7f820c009b20 comp rx=0 tx=0).stop 2023-11-08T08:21:12.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 -- 172.21.15.116:0/337898875 shutdown_connections 2023-11-08T08:21:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 --2- 172.21.15.116:0/337898875 >> v2:172.21.15.129:3300/0 conn(0x7f822006d4e0 0x7f82200ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 --2- 172.21.15.116:0/337898875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158d50 0x7f82200cc690 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 -- 172.21.15.116:0/337898875 >> 172.21.15.116:0/337898875 conn(0x7f82200cb540 msgr2=0x7f82200cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 -- 172.21.15.116:0/337898875 shutdown_connections 2023-11-08T08:21:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.448+0000 7f8226bd6700 1 -- 172.21.15.116:0/337898875 wait complete. 2023-11-08T08:21:12.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.449+0000 7f8226bd6700 1 Processor -- start 2023-11-08T08:21:12.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.449+0000 7f8226bd6700 1 -- start start 2023-11-08T08:21:12.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f8226bd6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822006d4e0 0x7f822014ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f8226bd6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 0x7f82200b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f8226bd6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8220158d50 0x7f82200708f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f8226bd6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220159ac0 con 0x7f8220158980 2023-11-08T08:21:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f8226bd6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220159da0 con 0x7f822006d4e0 2023-11-08T08:21:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f8226bd6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822015a080 con 0x7f8220158d50 2023-11-08T08:21:12.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 0x7f82200b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 0x7f82200b1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53590/0 (socket says 172.21.15.116:53590) 2023-11-08T08:21:12.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 -- 172.21.15.116:0/3249714761 learned_addr learned my addr 172.21.15.116:0/3249714761 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:12.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 -- 172.21.15.116:0/3249714761 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8220158d50 msgr2=0x7f82200708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8220158d50 0x7f82200708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 -- 172.21.15.116:0/3249714761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822006d4e0 msgr2=0x7f822014ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822006d4e0 0x7f822014ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.450+0000 7f821ffff700 1 -- 172.21.15.116:0/3249714761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0094a0 con 0x7f8220158980 2023-11-08T08:21:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.451+0000 7f821ffff700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 0x7f82200b1b60 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f820c014040 tx=0x7f820c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.451+0000 7f821d7fa700 1 -- 172.21.15.116:0/3249714761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c0038a0 con 0x7f8220158980 2023-11-08T08:21:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.451+0000 7f821d7fa700 1 -- 172.21.15.116:0/3249714761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c01e070 con 0x7f8220158980 2023-11-08T08:21:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.451+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82200cdb90 con 0x7f8220158980 2023-11-08T08:21:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.451+0000 7f821d7fa700 1 -- 172.21.15.116:0/3249714761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c003c40 con 0x7f8220158980 2023-11-08T08:21:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.451+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82200cd8b0 con 0x7f8220158980 2023-11-08T08:21:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.452+0000 7f821d7fa700 1 -- 172.21.15.116:0/3249714761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f820c00f050 con 0x7f8220158980 2023-11-08T08:21:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.452+0000 7f821d7fa700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82040682f0 0x7f820406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.453+0000 7f821d7fa700 1 -- 172.21.15.116:0/3249714761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f820c086dd0 con 0x7f8220158980 2023-11-08T08:21:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.453+0000 7f8224972700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82040682f0 0x7f820406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.453+0000 7f8224972700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82040682f0 0x7f820406a7a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f82140011e0 tx=0x7f82140079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 Processor -- start 2023-11-08T08:21:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 -- start start 2023-11-08T08:21:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 0x561ef3403a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3440ad0 0x561ef344a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef342c670 0x561ef3404b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef3381e30 con 0x561ef3449f60 2023-11-08T08:21:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef32ee190 con 0x561ef3440ad0 2023-11-08T08:21:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ef335ce00 con 0x561ef342c670 2023-11-08T08:21:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8224972700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 0x561ef3403a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.455+0000 7f8224972700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 0x561ef3403a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53606/0 (socket says 172.21.15.116:53606) 2023-11-08T08:21:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 -- 172.21.15.116:0/3485468953 learned_addr learned my addr 172.21.15.116:0/3485468953 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f821ffff700 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3440ad0 0x561ef344a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8225173700 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef342c670 0x561ef3404b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 -- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef342c670 msgr2=0x561ef3404b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef342c670 0x561ef3404b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 -- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3440ad0 msgr2=0x561ef344a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3440ad0 0x561ef344a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 -- 172.21.15.116:0/3485468953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef314a7e0 con 0x561ef3449f60 2023-11-08T08:21:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f8224972700 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 0x561ef3403a00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8214005dc0 tx=0x7f821400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdcf42bc6d281a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f820a7fc700 1 -- 172.21.15.116:0/3485468953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821401b070 con 0x561ef3449f60 2023-11-08T08:21:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f820a7fc700 1 -- 172.21.15.116:0/3485468953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214014430 con 0x561ef3449f60 2023-11-08T08:21:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.456+0000 7f820a7fc700 1 -- 172.21.15.116:0/3485468953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140147e0 con 0x561ef3449f60 2023-11-08T08:21:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 -- 172.21.15.116:0/3485468953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 msgr2=0x561ef3403a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 0x561ef3403a00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8214005dc0 tx=0x7f821400f000 comp rx=0 tx=0).stop 2023-11-08T08:21:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 -- 172.21.15.116:0/3485468953 shutdown_connections 2023-11-08T08:21:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef342c670 0x561ef3404b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3449f60 0x561ef3403a00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 --2- 172.21.15.116:0/3485468953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3440ad0 0x561ef344a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 -- 172.21.15.116:0/3485468953 >> 172.21.15.116:0/3485468953 conn(0x561ef33f9d30 msgr2=0x561ef3402f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 -- 172.21.15.116:0/3485468953 shutdown_connections 2023-11-08T08:21:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 -- 172.21.15.116:0/3485468953 wait complete. 2023-11-08T08:21:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.457+0000 7f8232d75b80 1 -- start start 2023-11-08T08:21:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 Processor -- start 2023-11-08T08:21:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3443910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 0x561ef342ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3449f60 0x561ef342f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef3356170 con 0x561ef3440ad0 2023-11-08T08:21:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef314b3e0 con 0x561ef342c670 2023-11-08T08:21:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ef32ee390 con 0x561ef3449f60 2023-11-08T08:21:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f8232d75b80 10 client.0 _ll_register_callbacks cb 0x561ef32f4450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f821ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 0x561ef342ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f821ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 0x561ef342ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53616/0 (socket says 172.21.15.116:53616) 2023-11-08T08:21:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.458+0000 7f821ffff700 1 -- 172.21.15.116:0/1977170122 learned_addr learned my addr 172.21.15.116:0/1977170122 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8224972700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3443910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8225173700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3449f60 0x561ef342f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f821ffff700 1 -- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3449f60 msgr2=0x561ef342f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f821ffff700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3449f60 0x561ef342f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f821ffff700 1 -- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 msgr2=0x561ef3443910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f821ffff700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3443910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f821ffff700 1 -- 172.21.15.116:0/1977170122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8214005a70 con 0x561ef3440ad0 2023-11-08T08:21:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f821ffff700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 0x561ef342ec50 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f820c0854d0 tx=0x7f820c05f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc970a106dfaff7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f81ff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f81ff7fe700 1 -- 172.21.15.116:0/1977170122 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c05eef0 con 0x561ef3440ad0 2023-11-08T08:21:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f81ff7fe700 1 -- 172.21.15.116:0/1977170122 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f820c05f050 con 0x561ef3440ad0 2023-11-08T08:21:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ef335a080 con 0x561ef3440ad0 2023-11-08T08:21:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f81ff7fe700 1 -- 172.21.15.116:0/1977170122 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f820c0530e0 con 0x561ef3440ad0 2023-11-08T08:21:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 10 client.6009 fetch_fsmap learned FSMap version 24 2023-11-08T08:21:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x561ef32370f0 con 0x561ef3440ad0 2023-11-08T08:21:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f81ff7fe700 1 -- 172.21.15.116:0/1977170122 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ef335a080 con 0x561ef3440ad0 2023-11-08T08:21:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f81ff7fe700 1 -- 172.21.15.116:0/1977170122 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f820c053240 con 0x561ef3440ad0 2023-11-08T08:21:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 10 client.6009 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:21:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 10 client.6009 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.b 2023-11-08T08:21:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 20 client.6009 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:12.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3427ee0 0x561ef345c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 4 client.6009 mds_command: new command op to 5781 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:21:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.459+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ef31a5f40 con 0x561ef3427ee0 2023-11-08T08:21:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.460+0000 7f8224972700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3427ee0 0x561ef345c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.460+0000 7f81ff7fe700 10 client.6009 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:21:12.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.460+0000 7f8224972700 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3427ee0 0x561ef345c6e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bab2976f5b3359d server_cookie=df02df83ef389d66 in_seq=0 out_seq=0 2023-11-08T08:21:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f81ff7fe700 1 -- 172.21.15.116:0/1977170122 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561ef31a5f40 con 0x561ef3427ee0 2023-11-08T08:21:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f81ff7fe700 10 client.6009 handle_command_reply: tid=0 2023-11-08T08:21:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 client.6009 shutdown 2023-11-08T08:21:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 msgr2=0x561ef342ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 0x561ef342ec50 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f820c0854d0 tx=0x7f820c05f430 comp rx=0 tx=0).stop 2023-11-08T08:21:12.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 shutdown_connections 2023-11-08T08:21:12.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3427ee0 0x561ef345c6e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3449f60 0x561ef342f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3443910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f81ff7fe700 0 client.6009 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:21:12.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1977170122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3440ad0 0x561ef342ec50 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 >> 172.21.15.116:0/1977170122 conn(0x561ef33f9d30 msgr2=0x561ef33feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 shutdown_connections 2023-11-08T08:21:12.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 1 -- 172.21.15.116:0/1977170122 wait complete. 2023-11-08T08:21:12.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.461+0000 7f8232d75b80 20 client.6009 trim_cache size 0 max 16384 2023-11-08T08:21:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.477+0000 7f8232d75b80 1 Processor -- start 2023-11-08T08:21:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.477+0000 7f8232d75b80 1 -- start start 2023-11-08T08:21:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3427ee0 0x561ef342ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 0x561ef33ff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef34a8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8232d75b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef33f9ad0 con 0x561ef342c670 2023-11-08T08:21:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef33f7030 con 0x561ef3427ee0 2023-11-08T08:21:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ef342ec50 con 0x561ef3440ad0 2023-11-08T08:21:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 0x561ef33ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 0x561ef33ff950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53630/0 (socket says 172.21.15.116:53630) 2023-11-08T08:21:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 -- 172.21.15.116:0/1099833610 learned_addr learned my addr 172.21.15.116:0/1099833610 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8224972700 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3427ee0 0x561ef342ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8225173700 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef34a8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 -- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 msgr2=0x561ef34a8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef34a8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 -- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3427ee0 msgr2=0x561ef342ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3427ee0 0x561ef342ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 -- 172.21.15.116:0/1099833610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef331d330 con 0x561ef342c670 2023-11-08T08:21:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f8224972700 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3427ee0 0x561ef342ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f821ffff700 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 0x561ef33ff950 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f820c056ff0 tx=0x7f820c05a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb91640d9e66f20 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f820affd700 1 -- 172.21.15.116:0/1099833610 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c055690 con 0x561ef342c670 2023-11-08T08:21:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f820affd700 1 -- 172.21.15.116:0/1099833610 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c05eef0 con 0x561ef342c670 2023-11-08T08:21:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.478+0000 7f820affd700 1 -- 172.21.15.116:0/1099833610 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c0530e0 con 0x561ef342c670 2023-11-08T08:21:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 -- 172.21.15.116:0/1099833610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 msgr2=0x561ef33ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 0x561ef33ff950 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f820c056ff0 tx=0x7f820c05a000 comp rx=0 tx=0).stop 2023-11-08T08:21:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 -- 172.21.15.116:0/1099833610 shutdown_connections 2023-11-08T08:21:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef34a8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef3427ee0 0x561ef342ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 --2- 172.21.15.116:0/1099833610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef342c670 0x561ef33ff950 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 -- 172.21.15.116:0/1099833610 >> 172.21.15.116:0/1099833610 conn(0x561ef33f9d30 msgr2=0x561ef344a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 -- 172.21.15.116:0/1099833610 shutdown_connections 2023-11-08T08:21:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 -- 172.21.15.116:0/1099833610 wait complete. 2023-11-08T08:21:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.479+0000 7f8232d75b80 1 -- start start 2023-11-08T08:21:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 Processor -- start 2023-11-08T08:21:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 0x561ef3442280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3463090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef342b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef34aedc0 con 0x561ef3427ee0 2023-11-08T08:21:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef33ff1e0 con 0x561ef342c670 2023-11-08T08:21:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ef3441df0 con 0x561ef3440ad0 2023-11-08T08:21:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8232d75b80 10 client.0 _ll_register_callbacks cb 0x561ef32f4450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8224972700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 0x561ef3442280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8224972700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 0x561ef3442280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53642/0 (socket says 172.21.15.116:53642) 2023-11-08T08:21:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.480+0000 7f8224972700 1 -- 172.21.15.116:0/2052434449 learned_addr learned my addr 172.21.15.116:0/2052434449 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8224972700 1 -- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 msgr2=0x561ef342b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8224972700 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef342b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8224972700 1 -- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 msgr2=0x561ef3463090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f821ffff700 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3463090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8224972700 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3463090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8224972700 1 -- 172.21.15.116:0/2052434449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef32370f0 con 0x561ef3427ee0 2023-11-08T08:21:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8224972700 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 0x561ef3442280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8214007870 tx=0x7f8214001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2470eef552ed1a7f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f81ff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f81ff7fe700 1 -- 172.21.15.116:0/2052434449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214027040 con 0x561ef3427ee0 2023-11-08T08:21:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f81ff7fe700 1 -- 172.21.15.116:0/2052434449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8214014930 con 0x561ef3427ee0 2023-11-08T08:21:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8214014430 con 0x561ef3427ee0 2023-11-08T08:21:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f81ff7fe700 1 -- 172.21.15.116:0/2052434449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f821402b5f0 con 0x561ef3427ee0 2023-11-08T08:21:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 10 client.6015 fetch_fsmap learned FSMap version 24 2023-11-08T08:21:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x561ef31ab220 con 0x561ef3427ee0 2023-11-08T08:21:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f81ff7fe700 1 -- 172.21.15.116:0/2052434449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214014430 con 0x561ef3427ee0 2023-11-08T08:21:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f81ff7fe700 1 -- 172.21.15.116:0/2052434449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f8214014af0 con 0x561ef3427ee0 2023-11-08T08:21:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 10 client.6015 fetch_fsmap finished waiting for FSMap version 24 2023-11-08T08:21:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 10 client.6015 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.b 2023-11-08T08:21:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 20 client.6015 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3449f60 0x561ef342bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 4 client.6015 mds_command: new command op to 5781 tid=0[{"prefix": "scrub status"}] 2023-11-08T08:21:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.481+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 --> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561ef31db090 con 0x561ef3449f60 2023-11-08T08:21:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f821ffff700 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3449f60 0x561ef342bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f81ff7fe700 10 client.6015 ms_handle_connect on v2:172.21.15.129:6833/409645101 2023-11-08T08:21:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f821ffff700 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3449f60 0x561ef342bac0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7ee77adb22f6c28 server_cookie=8b10b2063deee1ec in_seq=0 out_seq=0 2023-11-08T08:21:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f81ff7fe700 1 -- 172.21.15.116:0/2052434449 <== mds.0 v2:172.21.15.129:6833/409645101 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561ef31db090 con 0x561ef3449f60 2023-11-08T08:21:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f81ff7fe700 10 client.6015 handle_command_reply: tid=0 2023-11-08T08:21:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 client.6015 shutdown 2023-11-08T08:21:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 msgr2=0x561ef3442280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 0x561ef3442280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8214007870 tx=0x7f8214001080 comp rx=0 tx=0).stop 2023-11-08T08:21:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 shutdown_connections 2023-11-08T08:21:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:6833/409645101,v1:172.21.15.129:6834/409645101] conn(0x561ef3449f60 0x561ef342bac0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x561ef3440ad0 0x561ef342b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f81ff7fe700 0 client.6015 ms_handle_reset on v2:172.21.15.129:6833/409645101 2023-11-08T08:21:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ef3427ee0 0x561ef3442280 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 --2- 172.21.15.116:0/2052434449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561ef342c670 0x561ef3463090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 >> 172.21.15.116:0/2052434449 conn(0x561ef33f9d30 msgr2=0x561ef342ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 shutdown_connections 2023-11-08T08:21:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.482+0000 7f8232d75b80 1 -- 172.21.15.116:0/2052434449 wait complete. 2023-11-08T08:21:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8232d75b80 20 client.6015 trim_cache size 0 max 16384 2023-11-08T08:21:12.510 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2023-11-08T08:21:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2023-11-08T08:21:12.511 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82040682f0 msgr2=0x7f820406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82040682f0 0x7f820406a7a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f82140011e0 tx=0x7f82140079b0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 msgr2=0x7f82200b1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 0x7f82200b1b60 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f820c014040 tx=0x7f820c004750 comp rx=0 tx=0).stop 2023-11-08T08:21:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 shutdown_connections 2023-11-08T08:21:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8220158d50 0x7f82200708f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82040682f0 0x7f820406a7a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822006d4e0 0x7f822014ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 --2- 172.21.15.116:0/3249714761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8220158980 0x7f82200b1b60 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 >> 172.21.15.116:0/3249714761 conn(0x7f82200cb540 msgr2=0x7f82200b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 shutdown_connections 2023-11-08T08:21:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.483+0000 7f8226bd6700 1 -- 172.21.15.116:0/3249714761 wait complete. 2023-11-08T08:21:12.515 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-08T08:21:12.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:12.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.737+0000 7f33d3e95700 1 Processor -- start 2023-11-08T08:21:12.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.737+0000 7f33d3e95700 1 -- start start 2023-11-08T08:21:12.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d3e95700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f33cc070d90 0x7f33cc0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d3e95700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc151a50 con 0x7f33cc15e0b0 2023-11-08T08:21:12.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d3e95700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc15b440 con 0x7f33cc06a270 2023-11-08T08:21:12.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d3e95700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc071ac0 con 0x7f33cc070d90 2023-11-08T08:21:12.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d2432700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f33cc15e0b0 0x7f33cc15e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53200/0 (socket says 172.21.15.116:53200) 2023-11-08T08:21:12.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d2432700 1 -- 172.21.15.116:0/1144863662 learned_addr learned my addr 172.21.15.116:0/1144863662 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:12.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d1430700 1 --2- 172.21.15.116:0/1144863662 >> v2:172.21.15.129:3300/0 conn(0x7f33cc070d90 0x7f33cc0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352046206 0 0) 0x7f33cc151a50 con 0x7f33cc15e0b0 2023-11-08T08:21:12.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.738+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b80035b0 con 0x7f33cc15e0b0 2023-11-08T08:21:12.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225397624 0 0) 0x7f33b80035b0 con 0x7f33cc15e0b0 2023-11-08T08:21:12.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 >> v2:172.21.15.129:3300/0 conn(0x7f33cc070d90 msgr2=0x7f33cc0711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 --2- 172.21.15.116:0/1144863662 >> v2:172.21.15.129:3300/0 conn(0x7f33cc070d90 0x7f33cc0711e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 >> v1:172.21.15.129:6789/0 conn(0x7f33cc06a270 legacy=0x7f33cc06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc15adf0 con 0x7f33cc15e0b0 2023-11-08T08:21:12.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f33c40019f0 con 0x7f33cc15e0b0 2023-11-08T08:21:12.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 >> v1:172.21.15.116:6789/0 conn(0x7f33cc15e0b0 legacy=0x7f33cc15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33b8003a00 0x7f33b8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33b8004850 0x7f33b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8005200 0x7f33b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc071ac0 con 0x7f33b8004850 2023-11-08T08:21:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c080 con 0x7f33b8005200 2023-11-08T08:21:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c240 con 0x7f33b8003a00 2023-11-08T08:21:12.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33b8004850 0x7f33b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d2432700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8005200 0x7f33b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1c31700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33b8003a00 0x7f33b8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 -- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33b8003a00 msgr2=0x7f33b8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33b8003a00 0x7f33b8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 -- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8005200 msgr2=0x7f33b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8005200 0x7f33b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 -- 172.21.15.116:0/1144863662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80035b0 con 0x7f33b8004850 2023-11-08T08:21:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d2432700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8005200 0x7f33b800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1c31700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33b8003a00 0x7f33b8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d2432700 1 -- 172.21.15.116:0/1144863662 reap_dead start 2023-11-08T08:21:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d1430700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33b8004850 0x7f33b8004cc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009b60 tx=0x7f33b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaec823eef2002f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c800e6c0 con 0x7f33b8004850 2023-11-08T08:21:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c800e820 con 0x7f33b8004850 2023-11-08T08:21:12.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.739+0000 7f33d0c2f700 1 -- 172.21.15.116:0/1144863662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c800eb90 con 0x7f33b8004850 2023-11-08T08:21:12.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 -- 172.21.15.116:0/1144863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33b8004850 msgr2=0x7f33b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33b8004850 0x7f33b8004cc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009b60 tx=0x7f33b8005880 comp rx=0 tx=0).stop 2023-11-08T08:21:12.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 -- 172.21.15.116:0/1144863662 shutdown_connections 2023-11-08T08:21:12.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 --2- 172.21.15.116:0/1144863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33b8004850 0x7f33b8004cc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 -- 172.21.15.116:0/1144863662 >> 172.21.15.116:0/1144863662 conn(0x7f33cc06c010 msgr2=0x7f33cc0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 -- 172.21.15.116:0/1144863662 shutdown_connections 2023-11-08T08:21:12.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.740+0000 7f33d3e95700 1 -- 172.21.15.116:0/1144863662 wait complete. 2023-11-08T08:21:12.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.741+0000 7f33d3e95700 1 Processor -- start 2023-11-08T08:21:12.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.741+0000 7f33d3e95700 1 -- start start 2023-11-08T08:21:12.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8004850 0x7f33cc151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 0x7f33cc0c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33cc152b50 0x7f33cc152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c41b0 con 0x7f33cc0c3800 2023-11-08T08:21:12.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc151750 con 0x7f33b8004850 2023-11-08T08:21:12.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc153500 con 0x7f33cc152b50 2023-11-08T08:21:12.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 0x7f33cc0c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 0x7f33cc0c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53658/0 (socket says 172.21.15.116:53658) 2023-11-08T08:21:12.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 -- 172.21.15.116:0/3519387258 learned_addr learned my addr 172.21.15.116:0/3519387258 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:12.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1c31700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8004850 0x7f33cc151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d2432700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33cc152b50 0x7f33cc152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 -- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33cc152b50 msgr2=0x7f33cc152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33cc152b50 0x7f33cc152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 -- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8004850 msgr2=0x7f33cc151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8004850 0x7f33cc151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 -- 172.21.15.116:0/3519387258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc15aac0 con 0x7f33cc0c3800 2023-11-08T08:21:12.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1c31700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8004850 0x7f33cc151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d1430700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 0x7f33cc0c3c70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f33b800c050 tx=0x7f33c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c800e6c0 con 0x7f33cc0c3800 2023-11-08T08:21:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c8003ad0 con 0x7f33cc0c3800 2023-11-08T08:21:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc15b0d0 con 0x7f33cc0c3800 2023-11-08T08:21:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.742+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8003e80 con 0x7f33cc0c3800 2023-11-08T08:21:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.743+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc0b23e0 con 0x7f33cc0c3800 2023-11-08T08:21:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.743+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc0743f0 con 0x7f33cc0c3800 2023-11-08T08:21:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.744+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f33c8019030 con 0x7f33cc0c3800 2023-11-08T08:21:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.744+0000 7f33c27fc700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33b00682f0 0x7f33b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.744+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f33c8087380 con 0x7f33cc0c3800 2023-11-08T08:21:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.744+0000 7f33d1c31700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33b00682f0 0x7f33b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.745+0000 7f33d1c31700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33b00682f0 0x7f33b006a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f33bc000c00 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.748+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f33c8053be0 con 0x7f33cc0c3800 2023-11-08T08:21:12.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.969+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33cc0cd520 con 0x7f33cc0c3800 2023-11-08T08:21:12.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.970+0000 7f33c27fc700 1 -- 172.21.15.116:0/3519387258 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f33c8057200 con 0x7f33cc0c3800 2023-11-08T08:21:12.971 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:12.972 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:21:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33b00682f0 msgr2=0x7f33b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33b00682f0 0x7f33b006a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f33bc000c00 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2023-11-08T08:21:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 msgr2=0x7f33cc0c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 0x7f33cc0c3c70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f33b800c050 tx=0x7f33c80047c0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 shutdown_connections 2023-11-08T08:21:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33cc152b50 0x7f33cc152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33b00682f0 0x7f33b006a7a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33b8004850 0x7f33cc151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 --2- 172.21.15.116:0/3519387258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc0c3800 0x7f33cc0c3c70 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 >> 172.21.15.116:0/3519387258 conn(0x7f33cc06c010 msgr2=0x7f33cc0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 shutdown_connections 2023-11-08T08:21:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:12.972+0000 7f33d3e95700 1 -- 172.21.15.116:0/3519387258 wait complete. 2023-11-08T08:21:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:21:12.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2023-11-08T08:21:13.144 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2023-11-08T08:21:13.145 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-08T08:21:13.145 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2023-11-08T08:21:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 Processor -- start 2023-11-08T08:21:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 -- start start 2023-11-08T08:21:13.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4c003a60 0x562f9dd00760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dda7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 0x562f9dda80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9dc7c000 con 0x562f9dcd37c0 2023-11-08T08:21:13.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9dc0efe0 con 0x562f9dcd3350 2023-11-08T08:21:13.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd6bab50c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f9dd20250 con 0x7fcd4c003a60 2023-11-08T08:21:13.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd623a2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 0x562f9dda80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.190+0000 7fcd623a2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 0x562f9dda80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53678/0 (socket says 172.21.15.116:53678) 2023-11-08T08:21:13.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 -- 172.21.15.116:0/3899032301 learned_addr learned my addr 172.21.15.116:0/3899032301 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:13.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd61ba1700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4c003a60 0x562f9dd00760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd613a0700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dda7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 -- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4c003a60 msgr2=0x562f9dd00760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:13.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4c003a60 0x562f9dd00760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 -- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 msgr2=0x562f9dda7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:13.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dda7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 -- 172.21.15.116:0/3899032301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f9dc27360 con 0x562f9dcd37c0 2023-11-08T08:21:13.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd613a0700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dda7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:13.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd61ba1700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4c003a60 0x562f9dd00760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:13.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd623a2700 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 0x562f9dda80a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcd54001640 tx=0x7fcd5400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a674e2b5edd2766 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd53fff700 1 -- 172.21.15.116:0/3899032301 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54014040 con 0x562f9dcd37c0 2023-11-08T08:21:13.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd53fff700 1 -- 172.21.15.116:0/3899032301 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5400be60 con 0x562f9dcd37c0 2023-11-08T08:21:13.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd53fff700 1 -- 172.21.15.116:0/3899032301 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5401b670 con 0x562f9dcd37c0 2023-11-08T08:21:13.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3899032301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 msgr2=0x562f9dda80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 0x562f9dda80a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcd54001640 tx=0x7fcd5400f000 comp rx=0 tx=0).stop 2023-11-08T08:21:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3899032301 shutdown_connections 2023-11-08T08:21:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f9dcd37c0 0x562f9dda80a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd4c003a60 0x562f9dd00760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3899032301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dda7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3899032301 >> 172.21.15.116:0/3899032301 conn(0x562f9dd1c070 msgr2=0x562f9dd02d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:13.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3899032301 shutdown_connections 2023-11-08T08:21:13.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.191+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3899032301 wait complete. 2023-11-08T08:21:13.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.192+0000 7fcd6bab50c0 1 Processor -- start 2023-11-08T08:21:13.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.192+0000 7fcd6bab50c0 1 -- start start 2023-11-08T08:21:13.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd6bab50c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dd9f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd6bab50c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x562f9dcd37c0 0x562f9dd03fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd6bab50c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4c003a60 0x562f9dd04500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd6bab50c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9dd034e0 con 0x7fcd4c003a60 2023-11-08T08:21:13.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd6bab50c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9dd9c800 con 0x562f9dcd3350 2023-11-08T08:21:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd6bab50c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f9dda0030 con 0x562f9dcd37c0 2023-11-08T08:21:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4c003a60 0x562f9dd04500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4c003a60 0x562f9dd04500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53680/0 (socket says 172.21.15.116:53680) 2023-11-08T08:21:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 -- 172.21.15.116:0/3820654323 learned_addr learned my addr 172.21.15.116:0/3820654323 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd61ba1700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dd9f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd613a0700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x562f9dcd37c0 0x562f9dd03fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 -- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x562f9dcd37c0 msgr2=0x562f9dd03fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x562f9dcd37c0 0x562f9dd03fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 -- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 msgr2=0x562f9dd9f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f9dcd3350 0x562f9dd9f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 -- 172.21.15.116:0/3820654323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c001e10 con 0x7fcd4c003a60 2023-11-08T08:21:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd623a2700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4c003a60 0x562f9dd04500 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcd5401ba70 tx=0x7fcd54001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd52ffd700 1 -- 172.21.15.116:0/3820654323 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54014040 con 0x7fcd4c003a60 2023-11-08T08:21:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.193+0000 7fcd52ffd700 1 -- 172.21.15.116:0/3820654323 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd54001aa0 con 0x7fcd4c003a60 2023-11-08T08:21:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.194+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562f9ddafd00 con 0x7fcd4c003a60 2023-11-08T08:21:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.194+0000 7fcd52ffd700 1 -- 172.21.15.116:0/3820654323 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54028670 con 0x7fcd4c003a60 2023-11-08T08:21:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.194+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f9dc1a830 con 0x7fcd4c003a60 2023-11-08T08:21:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.195+0000 7fcd52ffd700 1 -- 172.21.15.116:0/3820654323 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd540289e0 con 0x7fcd4c003a60 2023-11-08T08:21:13.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.195+0000 7fcd52ffd700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd44066c10 0x7fcd440690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.195+0000 7fcd61ba1700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd44066c10 0x7fcd440690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.195+0000 7fcd52ffd700 1 -- 172.21.15.116:0/3820654323 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcd54087080 con 0x7fcd4c003a60 2023-11-08T08:21:13.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.195+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x562f9dd9cfc0 0x562f9dd9d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.195+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) v8 -- 0x562f9dd9d980 con 0x562f9dd9cfc0 2023-11-08T08:21:13.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd61ba1700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd44066c10 0x7fcd440690c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x562f9dda5d50 tx=0x7fcd5c007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd613a0700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x562f9dd9cfc0 0x562f9dd9d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd613a0700 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x562f9dd9cfc0 0x562f9dd9d440 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd613a0700 1 -- 172.21.15.116:0/3820654323 <== osd.7 v2:172.21.15.129:6816/121422 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7fcd58009850 con 0x562f9dd9cfc0 2023-11-08T08:21:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x562f9dd9cfc0 msgr2=0x562f9dd9d440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x562f9dd9cfc0 0x562f9dd9d440 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd44066c10 msgr2=0x7fcd440690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcd44066c10 0x7fcd440690c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x562f9dda5d50 tx=0x7fcd5c007040 comp rx=0 tx=0).stop 2023-11-08T08:21:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4c003a60 msgr2=0x562f9dd04500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 --2- 172.21.15.116:0/3820654323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4c003a60 0x562f9dd04500 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcd5401ba70 tx=0x7fcd54001120 comp rx=0 tx=0).stop 2023-11-08T08:21:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd623a2700 1 -- 172.21.15.116:0/3820654323 reap_dead start 2023-11-08T08:21:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 shutdown_connections 2023-11-08T08:21:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.196+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 >> 172.21.15.116:0/3820654323 conn(0x562f9dd1c070 msgr2=0x562f9ddab820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.197+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 shutdown_connections 2023-11-08T08:21:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.197+0000 7fcd6bab50c0 1 -- 172.21.15.116:0/3820654323 wait complete. 2023-11-08T08:21:13.214 INFO:teuthology.orchestra.run.smithi116.stdout:Yalphaparent_a 2023-11-08T08:21:13.214 DEBUG:teuthology.orchestra.run.smithi116:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-11-08T08:21:13.270 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:13.270 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1099511627777, 2023-11-08T08:21:13.270 INFO:teuthology.orchestra.run.smithi116.stdout: "ancestors": [ 2023-11-08T08:21:13.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:13.271 INFO:teuthology.orchestra.run.smithi116.stdout: "dirino": 1099511627776, 2023-11-08T08:21:13.271 INFO:teuthology.orchestra.run.smithi116.stdout: "dname": "alpha", 2023-11-08T08:21:13.271 INFO:teuthology.orchestra.run.smithi116.stdout: "version": 4 2023-11-08T08:21:13.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:13.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:13.272 INFO:teuthology.orchestra.run.smithi116.stdout: "dirino": 1, 2023-11-08T08:21:13.272 INFO:teuthology.orchestra.run.smithi116.stdout: "dname": "parent_a", 2023-11-08T08:21:13.272 INFO:teuthology.orchestra.run.smithi116.stdout: "version": 7 2023-11-08T08:21:13.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:13.273 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-11-08T08:21:13.273 INFO:teuthology.orchestra.run.smithi116.stdout: "pool": 8, 2023-11-08T08:21:13.273 INFO:teuthology.orchestra.run.smithi116.stdout: "old_pools": [] 2023-11-08T08:21:13.273 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:13.273 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:13.274 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-08T08:21:13.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:21:13.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:21:13.394 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-08T08:21:13.395 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:13.395 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:13.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.574+0000 7f0ffaa4c700 1 Processor -- start 2023-11-08T08:21:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.574+0000 7f0ffaa4c700 1 -- start start 2023-11-08T08:21:13.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.574+0000 7f0ffaa4c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ff406b7b0 0x7f0ff4072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.574+0000 7f0ffaa4c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40726c0 con 0x7f0ff40b0880 2023-11-08T08:21:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.574+0000 7f0ffaa4c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40b43f0 con 0x7f0ff40ad1f0 2023-11-08T08:21:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.574+0000 7f0ffaa4c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff406bc50 con 0x7f0ff406b7b0 2023-11-08T08:21:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff3fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0ff40b0880 0x7f0ff40aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53218/0 (socket says 172.21.15.116:53218) 2023-11-08T08:21:13.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff3fff700 1 -- 172.21.15.116:0/1544091178 learned_addr learned my addr 172.21.15.116:0/1544091178 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:13.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff37fe700 1 --2- 172.21.15.116:0/1544091178 >> v2:172.21.15.129:3300/0 conn(0x7f0ff406b7b0 0x7f0ff4072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419792858 0 0) 0x7f0ff40726c0 con 0x7f0ff40b0880 2023-11-08T08:21:13.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe00035b0 con 0x7f0ff40b0880 2023-11-08T08:21:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796879403 0 0) 0x7f0ff40b43f0 con 0x7f0ff40ad1f0 2023-11-08T08:21:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff40726c0 con 0x7f0ff40ad1f0 2023-11-08T08:21:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289505249 0 0) 0x7f0fe00035b0 con 0x7f0ff40b0880 2023-11-08T08:21:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff37fe700 1 -- 172.21.15.116:0/1544091178 >> v1:172.21.15.129:6789/0 conn(0x7f0ff40ad1f0 legacy=0x7f0ff40ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff37fe700 1 -- 172.21.15.116:0/1544091178 >> v1:172.21.15.116:6789/0 conn(0x7f0ff40b0880 legacy=0x7f0ff40aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff37fe700 1 -- 172.21.15.116:0/1544091178 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40b46d0 con 0x7f0ff406b7b0 2023-11-08T08:21:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff37fe700 1 --2- 172.21.15.116:0/1544091178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff406b7b0 0x7f0ff4072180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009840 tx=0x7f0fdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=202bbd5e2e8c7b78 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4001670 con 0x7f0ff406b7b0 2023-11-08T08:21:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc004030 con 0x7f0ff406b7b0 2023-11-08T08:21:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.575+0000 7f0ff2ffd700 1 -- 172.21.15.116:0/1544091178 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc0043c0 con 0x7f0ff406b7b0 2023-11-08T08:21:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/1544091178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff406b7b0 msgr2=0x7f0ff4072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/1544091178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff406b7b0 0x7f0ff4072180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009840 tx=0x7f0fdc009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/1544091178 shutdown_connections 2023-11-08T08:21:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/1544091178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff406b7b0 0x7f0ff4072180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/1544091178 >> 172.21.15.116:0/1544091178 conn(0x7f0ff40b2fc0 msgr2=0x7f0ff4071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/1544091178 shutdown_connections 2023-11-08T08:21:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.576+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/1544091178 wait complete. 2023-11-08T08:21:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.577+0000 7f0ffaa4c700 1 Processor -- start 2023-11-08T08:21:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.577+0000 7f0ffaa4c700 1 -- start start 2023-11-08T08:21:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ffaa4c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 0x7f0ff41535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ffaa4c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff40ad1f0 0x7f0ff4158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ffaa4c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0ff40b0880 0x7f0ff409cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ffaa4c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4157c40 con 0x7f0ff406b7b0 2023-11-08T08:21:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ffaa4c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff409d350 con 0x7f0ff40ad1f0 2023-11-08T08:21:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ffaa4c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff41572a0 con 0x7f0ff40b0880 2023-11-08T08:21:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 0x7f0ff41535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 0x7f0ff41535d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53692/0 (socket says 172.21.15.116:53692) 2023-11-08T08:21:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 -- 172.21.15.116:0/3783782756 learned_addr learned my addr 172.21.15.116:0/3783782756 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff37fe700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff40ad1f0 0x7f0ff4158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 -- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0ff40b0880 msgr2=0x7f0ff409cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff8fe9700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0ff40b0880 0x7f0ff409cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0ff40b0880 0x7f0ff409cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 -- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff40ad1f0 msgr2=0x7f0ff4158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff40ad1f0 0x7f0ff4158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff3fff700 1 -- 172.21.15.116:0/3783782756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc0094a0 con 0x7f0ff406b7b0 2023-11-08T08:21:13.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.578+0000 7f0ff37fe700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff40ad1f0 0x7f0ff4158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:13.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.579+0000 7f0ff3fff700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 0x7f0ff41535d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400c770 tx=0x7f0fe400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.579+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4007560 con 0x7f0ff406b7b0 2023-11-08T08:21:13.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.579+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe40076c0 con 0x7f0ff406b7b0 2023-11-08T08:21:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.579+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff40b4fe0 con 0x7f0ff406b7b0 2023-11-08T08:21:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.579+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4004a70 con 0x7f0ff406b7b0 2023-11-08T08:21:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.579+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff40b4d00 con 0x7f0ff406b7b0 2023-11-08T08:21:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.580+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff40b49b0 con 0x7f0ff406b7b0 2023-11-08T08:21:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.580+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0fe4010070 con 0x7f0ff406b7b0 2023-11-08T08:21:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.581+0000 7f0ff0ff9700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fd40682f0 0x7f0fd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.581+0000 7f0ff37fe700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fd40682f0 0x7f0fd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.581+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0fe4084e90 con 0x7f0ff406b7b0 2023-11-08T08:21:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.581+0000 7f0ff37fe700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fd40682f0 0x7f0fd406a7a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40cff70 tx=0x7f0fdc013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.585+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0fe4051690 con 0x7f0ff406b7b0 2023-11-08T08:21:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.768+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ff406f990 con 0x7f0ff406b7b0 2023-11-08T08:21:13.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.769+0000 7f0ff0ff9700 1 -- 172.21.15.116:0/3783782756 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+17909 (secure 0 0 0) 0x7f0fe4018070 con 0x7f0ff406b7b0 2023-11-08T08:21:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:13.770 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":44,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:34.014710+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fd40682f0 msgr2=0x7f0fd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fd40682f0 0x7f0fd406a7a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40cff70 tx=0x7f0fdc013040 comp rx=0 tx=0).stop 2023-11-08T08:21:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 msgr2=0x7f0ff41535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 0x7f0ff41535d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400c770 tx=0x7f0fe400cb30 comp rx=0 tx=0).stop 2023-11-08T08:21:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 shutdown_connections 2023-11-08T08:21:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0ff40b0880 0x7f0ff409cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff406b7b0 0x7f0ff41535d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0fd40682f0 0x7f0fd406a7a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 --2- 172.21.15.116:0/3783782756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff40ad1f0 0x7f0ff4158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 >> 172.21.15.116:0/3783782756 conn(0x7f0ff40b2fc0 msgr2=0x7f0ff40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 shutdown_connections 2023-11-08T08:21:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.771+0000 7f0ffaa4c700 1 -- 172.21.15.116:0/3783782756 wait complete. 2023-11-08T08:21:13.784 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:20:30.822342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:20:31.988683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:21:13.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:21:13.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.907+0000 7f338ceba700 1 Processor -- start 2023-11-08T08:21:13.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.907+0000 7f338ceba700 1 -- start start 2023-11-08T08:21:13.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338ceba700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338ceba700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338806ac00 con 0x7f33880cdf20 2023-11-08T08:21:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338ceba700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338815e0d0 con 0x7f338806a7b0 2023-11-08T08:21:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338ceba700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33880ce360 con 0x7f338815a4d0 2023-11-08T08:21:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f3386d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f33880cdf20 0x7f33880cb010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53224/0 (socket says 172.21.15.116:53224) 2023-11-08T08:21:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f3386d9d700 1 -- 172.21.15.116:0/4030760000 learned_addr learned my addr 172.21.15.116:0/4030760000 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338659c700 1 --2- 172.21.15.116:0/4030760000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724840795 0 0) 0x7f3380001490 con 0x7f33880cdf20 2023-11-08T08:21:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.908+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33700035b0 con 0x7f33880cdf20 2023-11-08T08:21:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794748520 0 0) 0x7f338815e0d0 con 0x7f338806a7b0 2023-11-08T08:21:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380001490 con 0x7f338806a7b0 2023-11-08T08:21:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338659c700 1 -- 172.21.15.116:0/4030760000 >> v1:172.21.15.129:6789/0 conn(0x7f338806a7b0 legacy=0x7f33880cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338659c700 1 -- 172.21.15.116:0/4030760000 >> v1:172.21.15.116:6789/0 conn(0x7f33880cdf20 legacy=0x7f33880cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338659c700 1 -- 172.21.15.116:0/4030760000 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33880ce560 con 0x7f338815a4d0 2023-11-08T08:21:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338659c700 1 --2- 172.21.15.116:0/4030760000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338806a270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f337c009870 tx=0x7f337c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0d51400e6d7b1fd server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c011830 con 0x7f338815a4d0 2023-11-08T08:21:13.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c011990 con 0x7f338815a4d0 2023-11-08T08:21:13.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338559a700 1 -- 172.21.15.116:0/4030760000 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c011d20 con 0x7f338815a4d0 2023-11-08T08:21:13.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338ceba700 1 -- 172.21.15.116:0/4030760000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 msgr2=0x7f338806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338ceba700 1 --2- 172.21.15.116:0/4030760000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338806a270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f337c009870 tx=0x7f337c009b80 comp rx=0 tx=0).stop 2023-11-08T08:21:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338ceba700 1 -- 172.21.15.116:0/4030760000 shutdown_connections 2023-11-08T08:21:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.909+0000 7f338ceba700 1 --2- 172.21.15.116:0/4030760000 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338806a270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.910+0000 7f338ceba700 1 -- 172.21.15.116:0/4030760000 >> 172.21.15.116:0/4030760000 conn(0x7f338806c010 msgr2=0x7f33880cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.910+0000 7f338ceba700 1 -- 172.21.15.116:0/4030760000 shutdown_connections 2023-11-08T08:21:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.910+0000 7f338ceba700 1 -- 172.21.15.116:0/4030760000 wait complete. 2023-11-08T08:21:13.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.911+0000 7f338ceba700 1 Processor -- start 2023-11-08T08:21:13.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.911+0000 7f338ceba700 1 -- start start 2023-11-08T08:21:13.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338ceba700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338806a7b0 0x7f33880b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338ceba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 0x7f33881587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338ceba700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338814e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338ceba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881591b0 con 0x7f33880cdf20 2023-11-08T08:21:13.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338ceba700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388159490 con 0x7f338806a7b0 2023-11-08T08:21:13.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338ceba700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338815aa60 con 0x7f338815a4d0 2023-11-08T08:21:13.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f3385d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 0x7f33881587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f3385d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 0x7f33881587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53694/0 (socket says 172.21.15.116:53694) 2023-11-08T08:21:13.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f3385d9b700 1 -- 172.21.15.116:0/336665959 learned_addr learned my addr 172.21.15.116:0/336665959 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:13.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f3386d9d700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338814e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.912+0000 7f338659c700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338806a7b0 0x7f33880b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3385d9b700 1 -- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 msgr2=0x7f338814e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3385d9b700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338814e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3385d9b700 1 -- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338806a7b0 msgr2=0x7f33880b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:13.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3385d9b700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338806a7b0 0x7f33880b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:13.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3385d9b700 1 -- 172.21.15.116:0/336665959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c0094a0 con 0x7f33880cdf20 2023-11-08T08:21:13.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3386d9d700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338814e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:13.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f338659c700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338806a7b0 0x7f33880b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:13.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3385d9b700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 0x7f33881587d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f337800c520 tx=0x7f337800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33780052d0 con 0x7f33880cdf20 2023-11-08T08:21:13.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3378005430 con 0x7f33880cdf20 2023-11-08T08:21:13.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338815e600 con 0x7f33880cdf20 2023-11-08T08:21:13.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33780038c0 con 0x7f33880cdf20 2023-11-08T08:21:13.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.913+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338815e360 con 0x7f33880cdf20 2023-11-08T08:21:13.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.914+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338815e8e0 con 0x7f33880cdf20 2023-11-08T08:21:13.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.914+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3378019030 con 0x7f33880cdf20 2023-11-08T08:21:13.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.915+0000 7f3376ffd700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33680682f0 0x7f336806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:13.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.915+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3378085160 con 0x7f33880cdf20 2023-11-08T08:21:13.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.915+0000 7f338659c700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33680682f0 0x7f336806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:13.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.915+0000 7f338659c700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33680682f0 0x7f336806a7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f337c0090b0 tx=0x7f337c01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:13.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:13.919+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3378051940 con 0x7f33880cdf20 2023-11-08T08:21:14.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.099+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f33880743f0 con 0x7f33880cdf20 2023-11-08T08:21:14.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.100+0000 7f3376ffd700 1 -- 172.21.15.116:0/336665959 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3378054f60 con 0x7f33880cdf20 2023-11-08T08:21:14.101 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:21:14.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.101+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33680682f0 msgr2=0x7f336806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.101+0000 7f338ceba700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33680682f0 0x7f336806a7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f337c0090b0 tx=0x7f337c01c040 comp rx=0 tx=0).stop 2023-11-08T08:21:14.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 msgr2=0x7f33881587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 0x7f33881587d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f337800c520 tx=0x7f337800c8e0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 shutdown_connections 2023-11-08T08:21:14.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f338815a4d0 0x7f338814e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33680682f0 0x7f336806a7a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338806a7b0 0x7f33880b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 --2- 172.21.15.116:0/336665959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33880cdf20 0x7f33881587d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 >> 172.21.15.116:0/336665959 conn(0x7f338806c010 msgr2=0x7f33881551e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:14.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 shutdown_connections 2023-11-08T08:21:14.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.102+0000 7f338ceba700 1 -- 172.21.15.116:0/336665959 wait complete. 2023-11-08T08:21:14.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:21:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.236+0000 7fcf3293c700 1 Processor -- start 2023-11-08T08:21:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.236+0000 7fcf3293c700 1 -- start start 2023-11-08T08:21:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf3293c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf2c15a5f0 0x7fcf2c0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf3293c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcf2c0cc200 0x7fcf2c0b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf3293c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c15e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf3293c700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c0b3190 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf3293c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c15a9c0 con 0x7fcf2c0cc200 2023-11-08T08:21:14.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf3293c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c7b80 con 0x7fcf2c0b2d10 2023-11-08T08:21:14.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf2bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf2c15a5f0 0x7fcf2c0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf2bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf2c15a5f0 0x7fcf2c0d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53708/0 (socket says 172.21.15.116:53708) 2023-11-08T08:21:14.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf2bfff700 1 -- 172.21.15.116:0/3992710727 learned_addr learned my addr 172.21.15.116:0/3992710727 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:14.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf2b7fe700 1 --2- 172.21.15.116:0/3992710727 >> v2:172.21.15.129:3300/0 conn(0x7fcf2c0cc200 0x7fcf2c0b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf30ed9700 1 --2- 172.21.15.116:0/3992710727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.237+0000 7fcf2bfff700 1 -- 172.21.15.116:0/3992710727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 msgr2=0x7fcf2c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2bfff700 1 --2- 172.21.15.116:0/3992710727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c15e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2bfff700 1 -- 172.21.15.116:0/3992710727 >> v2:172.21.15.129:3300/0 conn(0x7fcf2c0cc200 msgr2=0x7fcf2c0b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2bfff700 1 --2- 172.21.15.116:0/3992710727 >> v2:172.21.15.129:3300/0 conn(0x7fcf2c0cc200 0x7fcf2c0b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2bfff700 1 -- 172.21.15.116:0/3992710727 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c0cc990 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2b7fe700 1 --2- 172.21.15.116:0/3992710727 >> v2:172.21.15.129:3300/0 conn(0x7fcf2c0cc200 0x7fcf2c0b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2bfff700 1 --2- 172.21.15.116:0/3992710727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c0d0ad0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c06d8d0 tx=0x7fcf1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29c4c097c1a91d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2affd700 1 -- 172.21.15.116:0/3992710727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00b870 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2affd700 1 -- 172.21.15.116:0/3992710727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c00b9d0 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf2affd700 1 -- 172.21.15.116:0/3992710727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00bd60 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 -- 172.21.15.116:0/3992710727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 msgr2=0x7fcf2c0d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 --2- 172.21.15.116:0/3992710727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c0d0ad0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c06d8d0 tx=0x7fcf1c009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 -- 172.21.15.116:0/3992710727 shutdown_connections 2023-11-08T08:21:14.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 --2- 172.21.15.116:0/3992710727 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 --2- 172.21.15.116:0/3992710727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c0d0ad0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 --2- 172.21.15.116:0/3992710727 >> v2:172.21.15.129:3300/0 conn(0x7fcf2c0cc200 0x7fcf2c0b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 -- 172.21.15.116:0/3992710727 >> 172.21.15.116:0/3992710727 conn(0x7fcf2c06ea80 msgr2=0x7fcf2c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:14.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 -- 172.21.15.116:0/3992710727 shutdown_connections 2023-11-08T08:21:14.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.238+0000 7fcf3293c700 1 -- 172.21.15.116:0/3992710727 wait complete. 2023-11-08T08:21:14.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.239+0000 7fcf3293c700 1 Processor -- start 2023-11-08T08:21:14.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 -- start start 2023-11-08T08:21:14.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf2c0cc200 0x7fcf2c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c8810 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c8af0 con 0x7fcf2c0cc200 2023-11-08T08:21:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.240+0000 7fcf3293c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c8dd0 con 0x7fcf2c0b2d10 2023-11-08T08:21:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53716/0 (socket says 172.21.15.116:53716) 2023-11-08T08:21:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 -- 172.21.15.116:0/4283675424 learned_addr learned my addr 172.21.15.116:0/4283675424 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf2b7fe700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf2c0cc200 0x7fcf2c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf2bfff700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 -- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 msgr2=0x7fcf2c0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c0affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 -- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf2c0cc200 msgr2=0x7fcf2c1585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf2c0cc200 0x7fcf2c1585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 -- 172.21.15.116:0/4283675424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c009550 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf30ed9700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c158b00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400cc30 tx=0x7fcf2400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400aa50 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2400abb0 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c0ccf50 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24003a20 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.241+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c0ccc70 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.242+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c0cd230 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.243+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf2401f070 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.243+0000 7fcf28ff9700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf180682f0 0x7fcf1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.244+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcf240860a0 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.244+0000 7fcf2bfff700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf180682f0 0x7fcf1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.244+0000 7fcf2bfff700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf180682f0 0x7fcf1806a7a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c000c00 tx=0x7fcf1c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.247+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf24052890 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.426+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf2c0743f0 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.426+0000 7fcf28ff9700 1 -- 172.21.15.116:0/4283675424 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf2401d020 con 0x7fcf2c15a5f0 2023-11-08T08:21:14.428 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:21:14.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf180682f0 msgr2=0x7fcf1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf180682f0 0x7fcf1806a7a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c000c00 tx=0x7fcf1c008030 comp rx=0 tx=0).stop 2023-11-08T08:21:14.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 msgr2=0x7fcf2c158b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c158b00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400cc30 tx=0x7fcf2400cf40 comp rx=0 tx=0).stop 2023-11-08T08:21:14.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 shutdown_connections 2023-11-08T08:21:14.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf2c15a5f0 0x7fcf2c158b00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf2c0b2d10 0x7fcf2c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcf180682f0 0x7fcf1806a7a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 --2- 172.21.15.116:0/4283675424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf2c0cc200 0x7fcf2c1585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.428+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 >> 172.21.15.116:0/4283675424 conn(0x7fcf2c06ea80 msgr2=0x7fcf2c0c73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.429+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 shutdown_connections 2023-11-08T08:21:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.429+0000 7fcf3293c700 1 -- 172.21.15.116:0/4283675424 wait complete. 2023-11-08T08:21:14.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:21:14.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.564+0000 7f935bb09700 1 Processor -- start 2023-11-08T08:21:14.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.564+0000 7f935bb09700 1 -- start start 2023-11-08T08:21:14.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.565+0000 7f935bb09700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 0x7f93540b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.565+0000 7f935bb09700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f93540af0a0 0x7f935415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.565+0000 7f935bb09700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9354159ec0 con 0x7f93541599b0 2023-11-08T08:21:14.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.565+0000 7f935bb09700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93540b1f60 con 0x7f93540af0a0 2023-11-08T08:21:14.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.565+0000 7f935bb09700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935415d8a0 con 0x7f93540b1650 2023-11-08T08:21:14.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93598a5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 0x7f93540b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f93540af0a0 0x7f935415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93598a5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 0x7f93540b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:40966/0 (socket says 172.21.15.116:40966) 2023-11-08T08:21:14.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f93540af0a0 0x7f935415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:51360/0 (socket says 172.21.15.116:51360) 2023-11-08T08:21:14.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93590a4700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f93541599b0 0x7f935415d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:37860/0 (socket says 172.21.15.116:37860) 2023-11-08T08:21:14.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93598a5700 1 -- 172.21.15.116:0/402717416 learned_addr learned my addr 172.21.15.116:0/402717416 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:14.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 -- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 msgr2=0x7f93540b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 --2- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 0x7f93540b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 -- 172.21.15.116:0/402717416 >> v1:172.21.15.129:6789/0 conn(0x7f93541599b0 legacy=0x7f935415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 -- 172.21.15.116:0/402717416 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935415db80 con 0x7f93540af0a0 2023-11-08T08:21:14.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93588a3700 1 -- 172.21.15.116:0/402717416 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562268748 0 0) 0x7f9354159ec0 con 0x7f93541599b0 2023-11-08T08:21:14.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93598a5700 1 --2- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 0x7f93540b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:14.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f935a0a6700 1 --2- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93540af0a0 0x7f935415a350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f934c009d90 tx=0x7f934c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c66ce0cd92857fed server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93588a3700 1 -- 172.21.15.116:0/402717416 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c00e070 con 0x7f93540af0a0 2023-11-08T08:21:14.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93588a3700 1 -- 172.21.15.116:0/402717416 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c00bb20 con 0x7f93540af0a0 2023-11-08T08:21:14.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.566+0000 7f93588a3700 1 -- 172.21.15.116:0/402717416 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c012430 con 0x7f93540af0a0 2023-11-08T08:21:14.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 -- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93540af0a0 msgr2=0x7f935415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 --2- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93540af0a0 0x7f935415a350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f934c009d90 tx=0x7f934c00b330 comp rx=0 tx=0).stop 2023-11-08T08:21:14.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 -- 172.21.15.116:0/402717416 shutdown_connections 2023-11-08T08:21:14.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 --2- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93540af0a0 0x7f935415a350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 --2- 172.21.15.116:0/402717416 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540b1650 0x7f93540b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 -- 172.21.15.116:0/402717416 >> 172.21.15.116:0/402717416 conn(0x7f935406c010 msgr2=0x7f93540cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:14.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 -- 172.21.15.116:0/402717416 shutdown_connections 2023-11-08T08:21:14.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.567+0000 7f935bb09700 1 -- 172.21.15.116:0/402717416 wait complete. 2023-11-08T08:21:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.568+0000 7f935bb09700 1 Processor -- start 2023-11-08T08:21:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.568+0000 7f935bb09700 1 -- start start 2023-11-08T08:21:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935bb09700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540af0a0 0x7f9354145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935bb09700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 0x7f93541f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935bb09700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9354148480 0x7f93541f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935bb09700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93541492c0 con 0x7f93540b1650 2023-11-08T08:21:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935bb09700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93541f38a0 con 0x7f9354148480 2023-11-08T08:21:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935bb09700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935406a520 con 0x7f93540af0a0 2023-11-08T08:21:14.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f93590a4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 0x7f93541f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f93590a4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 0x7f93541f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53730/0 (socket says 172.21.15.116:53730) 2023-11-08T08:21:14.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f93590a4700 1 -- 172.21.15.116:0/149858565 learned_addr learned my addr 172.21.15.116:0/149858565 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:14.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f935a0a6700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9354148480 0x7f93541f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.569+0000 7f93598a5700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540af0a0 0x7f9354145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f93590a4700 1 -- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540af0a0 msgr2=0x7f9354145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f93590a4700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540af0a0 0x7f9354145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f93590a4700 1 -- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9354148480 msgr2=0x7f93541f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f93590a4700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9354148480 0x7f93541f3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f93590a4700 1 -- 172.21.15.116:0/149858565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350009550 con 0x7f93540b1650 2023-11-08T08:21:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f93590a4700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 0x7f93541f2e20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f934400c520 tx=0x7f934400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344004c80 con 0x7f93540b1650 2023-11-08T08:21:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344004de0 con 0x7f93540b1650 2023-11-08T08:21:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440072d0 con 0x7f93540b1650 2023-11-08T08:21:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c009a70 con 0x7f93540b1650 2023-11-08T08:21:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.570+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935415a890 con 0x7f93540b1650 2023-11-08T08:21:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.571+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935415de60 con 0x7f93540b1650 2023-11-08T08:21:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.572+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f934401a020 con 0x7f93540b1650 2023-11-08T08:21:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.572+0000 7f934a7fc700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93380682f0 0x7f933806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.572+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9344085e70 con 0x7f93540b1650 2023-11-08T08:21:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.572+0000 7f93598a5700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93380682f0 0x7f933806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.572+0000 7f93598a5700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93380682f0 0x7f933806a7a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f93540c3430 tx=0x7f9350008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.576+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9344052770 con 0x7f93540b1650 2023-11-08T08:21:14.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.753+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f935415aa20 con 0x7f93540b1650 2023-11-08T08:21:14.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.753+0000 7f934a7fc700 1 -- 172.21.15.116:0/149858565 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9344055d90 con 0x7f93540b1650 2023-11-08T08:21:14.755 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:21:14.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93380682f0 msgr2=0x7f933806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93380682f0 0x7f933806a7a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f93540c3430 tx=0x7f9350008040 comp rx=0 tx=0).stop 2023-11-08T08:21:14.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 msgr2=0x7f93541f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 0x7f93541f2e20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f934400c520 tx=0x7f934400c8e0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 shutdown_connections 2023-11-08T08:21:14.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f93380682f0 0x7f933806a7a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9354148480 0x7f93541f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f93540af0a0 0x7f9354145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 --2- 172.21.15.116:0/149858565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93540b1650 0x7f93541f2e20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 >> 172.21.15.116:0/149858565 conn(0x7f935406c010 msgr2=0x7f935415ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 shutdown_connections 2023-11-08T08:21:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.755+0000 7f935bb09700 1 -- 172.21.15.116:0/149858565 wait complete. 2023-11-08T08:21:14.768 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:21:14.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.890+0000 7f14f44fa700 1 Processor -- start 2023-11-08T08:21:14.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.890+0000 7f14f44fa700 1 -- start start 2023-11-08T08:21:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f44fa700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14ec0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f44fa700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f14ec0cdd50 0x7f14ec0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f44fa700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec15e0d0 con 0x7f14ec15a4d0 2023-11-08T08:21:14.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f44fa700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec0ce560 con 0x7f14ec0cdd50 2023-11-08T08:21:14.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f44fa700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec15e320 con 0x7f14ec06a980 2023-11-08T08:21:14.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f2296700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f14ec15a4d0 0x7f14ec06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53226/0 (socket says 172.21.15.116:53226) 2023-11-08T08:21:14.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f2296700 1 -- 172.21.15.116:0/2151662596 learned_addr learned my addr 172.21.15.116:0/2151662596 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:14.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765955929 0 0) 0x7f14ec15e0d0 con 0x7f14ec15a4d0 2023-11-08T08:21:14.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d80035b0 con 0x7f14ec15a4d0 2023-11-08T08:21:14.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f2a97700 1 --2- 172.21.15.116:0/2151662596 >> v2:172.21.15.129:3300/0 conn(0x7f14ec0cdd50 0x7f14ec0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.891+0000 7f14f1a95700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14ec0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873130317 0 0) 0x7f14d80035b0 con 0x7f14ec15a4d0 2023-11-08T08:21:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 msgr2=0x7f14ec0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14ec0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 >> v2:172.21.15.129:3300/0 conn(0x7f14ec0cdd50 msgr2=0x7f14ec0cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 --2- 172.21.15.116:0/2151662596 >> v2:172.21.15.129:3300/0 conn(0x7f14ec0cdd50 0x7f14ec0cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec15e600 con 0x7f14ec15a4d0 2023-11-08T08:21:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f14dc001670 con 0x7f14ec15a4d0 2023-11-08T08:21:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1a95700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14ec0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 >> v1:172.21.15.116:6789/0 conn(0x7f14ec15a4d0 legacy=0x7f14ec06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14d8003a00 0x7f14d8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8004430 0x7f14d80048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14d8005030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec0ce560 con 0x7f14d8003a00 2023-11-08T08:21:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec15e320 con 0x7f14d8004430 2023-11-08T08:21:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d800ba40 con 0x7f14ec06a980 2023-11-08T08:21:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14d8003a00 0x7f14d8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1a95700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8004430 0x7f14d80048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2a97700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14d8005030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 -- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 msgr2=0x7f14d8005030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14d8005030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 -- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8004430 msgr2=0x7f14d80048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:14.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8004430 0x7f14d80048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 -- 172.21.15.116:0/2151662596 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d80035b0 con 0x7f14d8003a00 2023-11-08T08:21:14.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2a97700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec06a980 0x7f14d8005030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:14.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f2296700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14d8003a00 0x7f14d8003e10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00c320 tx=0x7f14d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7167f57cfae54f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc00ce00 con 0x7f14d8003a00 2023-11-08T08:21:14.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc005960 con 0x7f14d8003a00 2023-11-08T08:21:14.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.892+0000 7f14f1294700 1 -- 172.21.15.116:0/2151662596 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc005d10 con 0x7f14d8003a00 2023-11-08T08:21:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f44fa700 1 -- 172.21.15.116:0/2151662596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14d8003a00 msgr2=0x7f14d8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f44fa700 1 --2- 172.21.15.116:0/2151662596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14d8003a00 0x7f14d8003e10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00c320 tx=0x7f14d8005880 comp rx=0 tx=0).stop 2023-11-08T08:21:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f2a97700 1 -- 172.21.15.116:0/2151662596 reap_dead start 2023-11-08T08:21:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f44fa700 1 -- 172.21.15.116:0/2151662596 shutdown_connections 2023-11-08T08:21:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f44fa700 1 -- 172.21.15.116:0/2151662596 >> 172.21.15.116:0/2151662596 conn(0x7f14ec06c010 msgr2=0x7f14ec0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f44fa700 1 -- 172.21.15.116:0/2151662596 shutdown_connections 2023-11-08T08:21:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.893+0000 7f14f44fa700 1 -- 172.21.15.116:0/2151662596 wait complete. 2023-11-08T08:21:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.894+0000 7f14f44fa700 1 Processor -- start 2023-11-08T08:21:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.894+0000 7f14f44fa700 1 -- start start 2023-11-08T08:21:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8003a00 0x7f14ec151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 0x7f14ec0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec15d090 0x7f14ec15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec0c95d0 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec15da70 con 0x7f14d8003a00 2023-11-08T08:21:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec0c8a50 con 0x7f14ec15d090 2023-11-08T08:21:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 0x7f14ec0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 0x7f14ec0c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53756/0 (socket says 172.21.15.116:53756) 2023-11-08T08:21:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 -- 172.21.15.116:0/1704936820 learned_addr learned my addr 172.21.15.116:0/1704936820 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f2296700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8003a00 0x7f14ec151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f2a97700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec15d090 0x7f14ec15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 -- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec15d090 msgr2=0x7f14ec15d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec15d090 0x7f14ec15d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 -- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8003a00 msgr2=0x7f14ec151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8003a00 0x7f14ec151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 -- 172.21.15.116:0/1704936820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec15a8a0 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f1a95700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 0x7f14ec0c80e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009630 tx=0x7f14e8009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80146d0 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e8014830 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8014bc0 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.895+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec15e8e0 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.896+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec0c9010 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.896+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec0743f0 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.897+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f14e8024070 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.897+0000 7f14e2ffd700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14d00682f0 0x7f14d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:14.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.897+0000 7f14f2296700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14d00682f0 0x7f14d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:14.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.897+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f14e8085e30 con 0x7f14ec0c7ae0 2023-11-08T08:21:14.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.898+0000 7f14f2296700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14d00682f0 0x7f14d006a7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f14dc008fd0 tx=0x7f14dc00a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:14.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:14.902+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f14e8052690 con 0x7f14ec0c7ae0 2023-11-08T08:21:15.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.081+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14ec159cf0 con 0x7f14ec0c7ae0 2023-11-08T08:21:15.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.081+0000 7f14e2ffd700 1 -- 172.21.15.116:0/1704936820 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f14e8055cb0 con 0x7f14ec0c7ae0 2023-11-08T08:21:15.083 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:21:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14d00682f0 msgr2=0x7f14d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14d00682f0 0x7f14d006a7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f14dc008fd0 tx=0x7f14dc00a000 comp rx=0 tx=0).stop 2023-11-08T08:21:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 msgr2=0x7f14ec0c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 0x7f14ec0c80e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009630 tx=0x7f14e8009940 comp rx=0 tx=0).stop 2023-11-08T08:21:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 shutdown_connections 2023-11-08T08:21:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14ec15d090 0x7f14ec15d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f14d00682f0 0x7f14d006a7a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d8003a00 0x7f14ec151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 --2- 172.21.15.116:0/1704936820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ec0c7ae0 0x7f14ec0c80e0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 >> 172.21.15.116:0/1704936820 conn(0x7f14ec06c010 msgr2=0x7f14ec154640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 shutdown_connections 2023-11-08T08:21:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.083+0000 7f14f44fa700 1 -- 172.21.15.116:0/1704936820 wait complete. 2023-11-08T08:21:15.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:21:15.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.215+0000 7feb0eda8700 1 Processor -- start 2023-11-08T08:21:15.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0eda8700 1 -- start start 2023-11-08T08:21:15.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0eda8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb080cf710 0x7feb080ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:15.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0eda8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb08071890 0x7feb08071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:15.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0eda8700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0815cc10 con 0x7feb0806a270 2023-11-08T08:21:15.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0eda8700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0806c9b0 con 0x7feb08071890 2023-11-08T08:21:15.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0eda8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0806cc90 con 0x7feb080cf710 2023-11-08T08:21:15.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb07fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb08071890 0x7feb08071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:15.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0d345700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feb0806a270 0x7feb080b1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53238/0 (socket says 172.21.15.116:53238) 2023-11-08T08:21:15.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.216+0000 7feb0d345700 1 -- 172.21.15.116:0/2573442287 learned_addr learned my addr 172.21.15.116:0/2573442287 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:15.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb0cb44700 1 --2- 172.21.15.116:0/2573442287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb080cf710 0x7feb080ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:15.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb07fff700 1 -- 172.21.15.116:0/2573442287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb080cf710 msgr2=0x7feb080ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:15.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb07fff700 1 --2- 172.21.15.116:0/2573442287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb080cf710 0x7feb080ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb07fff700 1 -- 172.21.15.116:0/2573442287 >> v1:172.21.15.116:6789/0 conn(0x7feb0806a270 legacy=0x7feb080b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb077fe700 1 -- 172.21.15.116:0/2573442287 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547187645 0 0) 0x7feb0815cc10 con 0x7feb0806a270 2023-11-08T08:21:15.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb07fff700 1 -- 172.21.15.116:0/2573442287 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0806cf70 con 0x7feb08071890 2023-11-08T08:21:15.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb07fff700 1 --2- 172.21.15.116:0/2573442287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb08071890 0x7feb08071220 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009840 tx=0x7feaf4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f660ac0cb024dd6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:15.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb077fe700 1 -- 172.21.15.116:0/2573442287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4011830 con 0x7feb08071890 2023-11-08T08:21:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb077fe700 1 -- 172.21.15.116:0/2573442287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf4011990 con 0x7feb08071890 2023-11-08T08:21:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.217+0000 7feb077fe700 1 -- 172.21.15.116:0/2573442287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4011d20 con 0x7feb08071890 2023-11-08T08:21:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 -- 172.21.15.116:0/2573442287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb08071890 msgr2=0x7feb08071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 --2- 172.21.15.116:0/2573442287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb08071890 0x7feb08071220 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009840 tx=0x7feaf4009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:15.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 -- 172.21.15.116:0/2573442287 shutdown_connections 2023-11-08T08:21:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 --2- 172.21.15.116:0/2573442287 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb080cf710 0x7feb080ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 --2- 172.21.15.116:0/2573442287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb08071890 0x7feb08071220 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 -- 172.21.15.116:0/2573442287 >> 172.21.15.116:0/2573442287 conn(0x7feb0806c010 msgr2=0x7feb080cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 -- 172.21.15.116:0/2573442287 shutdown_connections 2023-11-08T08:21:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.218+0000 7feb0eda8700 1 -- 172.21.15.116:0/2573442287 wait complete. 2023-11-08T08:21:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.219+0000 7feb0eda8700 1 Processor -- start 2023-11-08T08:21:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.219+0000 7feb0eda8700 1 -- start start 2023-11-08T08:21:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0806a270 0x7feb080cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb08071890 0x7feb081f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 0x7feb081e66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0815f670 con 0x7feb080cf710 2023-11-08T08:21:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080ccb20 con 0x7feb08071890 2023-11-08T08:21:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0815e9f0 con 0x7feb0806a270 2023-11-08T08:21:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 0x7feb081e66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0cb44700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0806a270 0x7feb080cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 0x7feb081e66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53776/0 (socket says 172.21.15.116:53776) 2023-11-08T08:21:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 -- 172.21.15.116:0/1804924923 learned_addr learned my addr 172.21.15.116:0/1804924923 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb07fff700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb08071890 0x7feb081f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 -- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0806a270 msgr2=0x7feb080cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0806a270 0x7feb080cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 -- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb08071890 msgr2=0x7feb081f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb08071890 0x7feb081f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 -- 172.21.15.116:0/1804924923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf40094a0 con 0x7feb080cf710 2023-11-08T08:21:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0d345700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 0x7feb081e66a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7feb0000cca0 tx=0x7feb0000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000ab70 con 0x7feb080cf710 2023-11-08T08:21:15.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0000acd0 con 0x7feb080cf710 2023-11-08T08:21:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0806d530 con 0x7feb080cf710 2023-11-08T08:21:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.220+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00007310 con 0x7feb080cf710 2023-11-08T08:21:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.221+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0806d250 con 0x7feb080cf710 2023-11-08T08:21:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.221+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0806a6f0 con 0x7feb080cf710 2023-11-08T08:21:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.222+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb0001a030 con 0x7feb080cf710 2023-11-08T08:21:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.222+0000 7feb057fa700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaec0682f0 0x7feaec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.222+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feb00085990 con 0x7feb080cf710 2023-11-08T08:21:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.222+0000 7feb0cb44700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaec0682f0 0x7feaec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:15.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.223+0000 7feb0cb44700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaec0682f0 0x7feaec06a7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feafc006b20 tx=0x7feafc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:15.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.227+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb000521f0 con 0x7feb080cf710 2023-11-08T08:21:15.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.423+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feb0815f7b0 con 0x7feb080cf710 2023-11-08T08:21:15.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.423+0000 7feb057fa700 1 -- 172.21.15.116:0/1804924923 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7feb0815f7b0 con 0x7feb080cf710 2023-11-08T08:21:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaec0682f0 msgr2=0x7feaec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaec0682f0 0x7feaec06a7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feafc006b20 tx=0x7feafc008040 comp rx=0 tx=0).stop 2023-11-08T08:21:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 msgr2=0x7feb081e66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 0x7feb081e66a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7feb0000cca0 tx=0x7feb0000a060 comp rx=0 tx=0).stop 2023-11-08T08:21:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 shutdown_connections 2023-11-08T08:21:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb080cf710 0x7feb081e66a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7feb0806a270 0x7feb080cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7feaec0682f0 0x7feaec06a7a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 --2- 172.21.15.116:0/1804924923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb08071890 0x7feb081f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 >> 172.21.15.116:0/1804924923 conn(0x7feb0806c010 msgr2=0x7feb080cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 shutdown_connections 2023-11-08T08:21:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:15.425+0000 7feb0eda8700 1 -- 172.21.15.116:0/1804924923 wait complete. 2023-11-08T08:21:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:21:15.438 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-08T08:21:15.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:15.439 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:15.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[142354]: fuse finished with error 0 and tester_r 0 2023-11-08T08:21:15.479 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:21:15.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:15.480 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-08T08:21:15.645 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:15.645 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:15.666 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:15.667 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:21:15.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:15.668 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-08T08:21:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:21:15.727 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-11-08T08:21:15.728 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:15.728 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:15.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:21:15.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:21:15.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:15.749 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-08T08:21:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:21:15.805 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-11-08T08:21:15.806 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:15.806 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:15.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:15.828 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:21:15.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:15.829 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-08T08:21:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:21:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-11-08T08:21:15.889 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:16.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.007+0000 7f2a2b1c7700 1 Processor -- start 2023-11-08T08:21:16.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.007+0000 7f2a2b1c7700 1 -- start start 2023-11-08T08:21:16.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a2b1c7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a2406a270 0x7f2a2406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a2b1c7700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2a2406e570 0x7f2a24070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a2b1c7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2406ab80 con 0x7f2a2415cc80 2023-11-08T08:21:16.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a2b1c7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240b2490 con 0x7f2a2406a270 2023-11-08T08:21:16.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a2b1c7700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2406e990 con 0x7f2a2406e570 2023-11-08T08:21:16.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a28f63700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a2406a270 0x7f2a2406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a23fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2a2415cc80 0x7f2a2406de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53242/0 (socket says 172.21.15.116:53242) 2023-11-08T08:21:16.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.008+0000 7f2a23fff700 1 -- 172.21.15.116:0/3484571230 learned_addr learned my addr 172.21.15.116:0/3484571230 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:16.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a28f63700 1 -- 172.21.15.116:0/3484571230 >> v2:172.21.15.129:3300/0 conn(0x7f2a2406e570 msgr2=0x7f2a24070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a28f63700 1 --2- 172.21.15.116:0/3484571230 >> v2:172.21.15.129:3300/0 conn(0x7f2a2406e570 0x7f2a24070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a28f63700 1 -- 172.21.15.116:0/3484571230 >> v1:172.21.15.116:6789/0 conn(0x7f2a2415cc80 legacy=0x7f2a2406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a237fe700 1 -- 172.21.15.116:0/3484571230 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924717431 0 0) 0x7f2a2406ab80 con 0x7f2a2415cc80 2023-11-08T08:21:16.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a28f63700 1 -- 172.21.15.116:0/3484571230 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240b1810 con 0x7f2a2406a270 2023-11-08T08:21:16.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a28f63700 1 --2- 172.21.15.116:0/3484571230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2406a270 0x7f2a2406a640 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2a2406ee30 tx=0x7f2a18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e61e033de05cbd9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a237fe700 1 -- 172.21.15.116:0/3484571230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18011830 con 0x7f2a2406a270 2023-11-08T08:21:16.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a237fe700 1 -- 172.21.15.116:0/3484571230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18011990 con 0x7f2a2406a270 2023-11-08T08:21:16.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a237fe700 1 -- 172.21.15.116:0/3484571230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18011d20 con 0x7f2a2406a270 2023-11-08T08:21:16.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/3484571230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2406a270 msgr2=0x7f2a2406a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/3484571230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2406a270 0x7f2a2406a640 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2a2406ee30 tx=0x7f2a18009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:16.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/3484571230 shutdown_connections 2023-11-08T08:21:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/3484571230 >> v2:172.21.15.129:3300/0 conn(0x7f2a2406e570 0x7f2a24070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/3484571230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2406a270 0x7f2a2406a640 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.009+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/3484571230 >> 172.21.15.116:0/3484571230 conn(0x7f2a2406c010 msgr2=0x7f2a240cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.010+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/3484571230 shutdown_connections 2023-11-08T08:21:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.010+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/3484571230 wait complete. 2023-11-08T08:21:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.011+0000 7f2a2b1c7700 1 Processor -- start 2023-11-08T08:21:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.011+0000 7f2a2b1c7700 1 -- start start 2023-11-08T08:21:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a2406a270 0x7f2a240b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a2406e570 0x7f2a241e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 0x7f2a2415c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240b43d0 con 0x7f2a2415cc80 2023-11-08T08:21:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240b3790 con 0x7f2a2406e570 2023-11-08T08:21:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240b3a30 con 0x7f2a2406a270 2023-11-08T08:21:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 0x7f2a2415c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 0x7f2a2415c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53790/0 (socket says 172.21.15.116:53790) 2023-11-08T08:21:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 -- 172.21.15.116:0/4103520536 learned_addr learned my addr 172.21.15.116:0/4103520536 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a23fff700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a2406e570 0x7f2a241e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a28f63700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a2406a270 0x7f2a240b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 -- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a2406a270 msgr2=0x7f2a240b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a2406a270 0x7f2a240b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 -- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a2406e570 msgr2=0x7f2a241e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a2406e570 0x7f2a241e5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 -- 172.21.15.116:0/4103520536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180094a0 con 0x7f2a2415cc80 2023-11-08T08:21:16.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a23fff700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a2406e570 0x7f2a241e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:16.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a29764700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 0x7f2a2415c1a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00b970 tx=0x7f2a1c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c00c9b0 con 0x7f2a2415cc80 2023-11-08T08:21:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c00cb30 con 0x7f2a2415cc80 2023-11-08T08:21:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a240b1dd0 con 0x7f2a2415cc80 2023-11-08T08:21:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c014440 con 0x7f2a2415cc80 2023-11-08T08:21:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.012+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a240b1af0 con 0x7f2a2415cc80 2023-11-08T08:21:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.013+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a240b20b0 con 0x7f2a2415cc80 2023-11-08T08:21:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.014+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a1c01b030 con 0x7f2a2415cc80 2023-11-08T08:21:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.014+0000 7f2a217fa700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a080682f0 0x7f2a0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.014+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2a1c085910 con 0x7f2a2415cc80 2023-11-08T08:21:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.014+0000 7f2a28f63700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a080682f0 0x7f2a0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.014+0000 7f2a28f63700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a080682f0 0x7f2a0806a7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2a18000c00 tx=0x7f2a18008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.018+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a1c052170 con 0x7f2a2415cc80 2023-11-08T08:21:16.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.238+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a240743f0 con 0x7f2a2415cc80 2023-11-08T08:21:16.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.238+0000 7f2a217fa700 1 -- 172.21.15.116:0/4103520536 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f2a1c019090 con 0x7f2a2415cc80 2023-11-08T08:21:16.240 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:21:16.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a080682f0 msgr2=0x7f2a0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a080682f0 0x7f2a0806a7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2a18000c00 tx=0x7f2a18008030 comp rx=0 tx=0).stop 2023-11-08T08:21:16.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 msgr2=0x7f2a2415c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 0x7f2a2415c1a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00b970 tx=0x7f2a1c00bd30 comp rx=0 tx=0).stop 2023-11-08T08:21:16.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 shutdown_connections 2023-11-08T08:21:16.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2415cc80 0x7f2a2415c1a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a2406a270 0x7f2a240b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a080682f0 0x7f2a0806a7a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 --2- 172.21.15.116:0/4103520536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a2406e570 0x7f2a241e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.240+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 >> 172.21.15.116:0/4103520536 conn(0x7f2a2406c010 msgr2=0x7f2a240ad660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:16.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.241+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 shutdown_connections 2023-11-08T08:21:16.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.241+0000 7f2a2b1c7700 1 -- 172.21.15.116:0/4103520536 wait complete. 2023-11-08T08:21:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:21:16.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:16.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.372+0000 7fc035b8e700 1 Processor -- start 2023-11-08T08:21:16.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.372+0000 7fc035b8e700 1 -- start start 2023-11-08T08:21:16.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc035b8e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc0300b1770 0x7fc0300b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc035b8e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03015e010 con 0x7fc030142db0 2023-11-08T08:21:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc035b8e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03013fc90 con 0x7fc0300ce040 2023-11-08T08:21:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc035b8e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03013f480 con 0x7fc0300b1770 2023-11-08T08:21:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02effd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc030142db0 0x7fc03013ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53258/0 (socket says 172.21.15.116:53258) 2023-11-08T08:21:16.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02effd700 1 -- 172.21.15.116:0/2112312181 learned_addr learned my addr 172.21.15.116:0/2112312181 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879293625 0 0) 0x7fc020001110 con 0x7fc030142db0 2023-11-08T08:21:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02ffff700 1 --2- 172.21.15.116:0/2112312181 >> v2:172.21.15.129:3300/0 conn(0x7fc0300b1770 0x7fc0300b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01c0035b0 con 0x7fc030142db0 2023-11-08T08:21:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344458756 0 0) 0x7fc03013fc90 con 0x7fc0300ce040 2023-11-08T08:21:16.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc020001110 con 0x7fc0300ce040 2023-11-08T08:21:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 31181538 0 0) 0x7fc01c0035b0 con 0x7fc030142db0 2023-11-08T08:21:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 >> v2:172.21.15.129:3300/0 conn(0x7fc0300b1770 msgr2=0x7fc0300b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 --2- 172.21.15.116:0/2112312181 >> v2:172.21.15.129:3300/0 conn(0x7fc0300b1770 0x7fc0300b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 >> v1:172.21.15.129:6789/0 conn(0x7fc0300ce040 legacy=0x7fc03015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300b20d0 con 0x7fc030142db0 2023-11-08T08:21:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.373+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fc020001800 con 0x7fc030142db0 2023-11-08T08:21:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 >> v1:172.21.15.116:6789/0 conn(0x7fc030142db0 legacy=0x7fc03013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01c003be0 0x7fc01c003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c004850 0x7fc01c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01c005200 0x7fc01c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03013f480 con 0x7fc01c003be0 2023-11-08T08:21:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c00c0e0 con 0x7fc01c004850 2023-11-08T08:21:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c00c270 con 0x7fc01c005200 2023-11-08T08:21:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01c003be0 0x7fc01c003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02effd700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c004850 0x7fc01c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02ffff700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01c005200 0x7fc01c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 -- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01c005200 msgr2=0x7fc01c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01c005200 0x7fc01c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 -- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c004850 msgr2=0x7fc01c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c004850 0x7fc01c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 -- 172.21.15.116:0/2112312181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0035b0 con 0x7fc01c003be0 2023-11-08T08:21:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02ffff700 1 -- 172.21.15.116:0/2112312181 reap_dead start 2023-11-08T08:21:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01c003be0 0x7fc01c003fd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc018007fd0 tx=0x7fc01c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919aebe8dd82cd40 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018001430 con 0x7fc01c003be0 2023-11-08T08:21:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0180050d0 con 0x7fc01c003be0 2023-11-08T08:21:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.374+0000 7fc02e7fc700 1 -- 172.21.15.116:0/2112312181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018005480 con 0x7fc01c003be0 2023-11-08T08:21:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 -- 172.21.15.116:0/2112312181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01c003be0 msgr2=0x7fc01c003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01c003be0 0x7fc01c003fd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc018007fd0 tx=0x7fc01c005880 comp rx=0 tx=0).stop 2023-11-08T08:21:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 -- 172.21.15.116:0/2112312181 shutdown_connections 2023-11-08T08:21:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 --2- 172.21.15.116:0/2112312181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01c003be0 0x7fc01c003fd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 -- 172.21.15.116:0/2112312181 >> 172.21.15.116:0/2112312181 conn(0x7fc03006c010 msgr2=0x7fc0300cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:16.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 -- 172.21.15.116:0/2112312181 shutdown_connections 2023-11-08T08:21:16.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.375+0000 7fc035b8e700 1 -- 172.21.15.116:0/2112312181 wait complete. 2023-11-08T08:21:16.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.376+0000 7fc035b8e700 1 Processor -- start 2023-11-08T08:21:16.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.376+0000 7fc035b8e700 1 -- start start 2023-11-08T08:21:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc035b8e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c003be0 0x7fc0300c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc035b8e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 0x7fc0300c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc035b8e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0300c7570 0x7fc030150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc035b8e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300c79e0 con 0x7fc03006a950 2023-11-08T08:21:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc035b8e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300c7bd0 con 0x7fc01c003be0 2023-11-08T08:21:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc035b8e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300c7eb0 con 0x7fc0300c7570 2023-11-08T08:21:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 0x7fc0300c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 0x7fc0300c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53804/0 (socket says 172.21.15.116:53804) 2023-11-08T08:21:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 -- 172.21.15.116:0/1695094312 learned_addr learned my addr 172.21.15.116:0/1695094312 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02ffff700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0300c7570 0x7fc030150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c003be0 0x7fc0300c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 -- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0300c7570 msgr2=0x7fc030150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0300c7570 0x7fc030150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 -- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c003be0 msgr2=0x7fc0300c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c003be0 0x7fc0300c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.377+0000 7fc02effd700 1 -- 172.21.15.116:0/1695094312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03013f9b0 con 0x7fc03006a950 2023-11-08T08:21:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.378+0000 7fc02effd700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 0x7fc0300c7030 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc020007290 tx=0x7fc020007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.378+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02000f6c0 con 0x7fc03006a950 2023-11-08T08:21:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.378+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02000f860 con 0x7fc03006a950 2023-11-08T08:21:16.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.378+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02000fc10 con 0x7fc03006a950 2023-11-08T08:21:16.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.378+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03013f6d0 con 0x7fc03006a950 2023-11-08T08:21:16.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.378+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03015d100 con 0x7fc03006a950 2023-11-08T08:21:16.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.379+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03006d270 con 0x7fc03006a950 2023-11-08T08:21:16.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.379+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc02000fd70 con 0x7fc03006a950 2023-11-08T08:21:16.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.380+0000 7fc027fff700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0100682f0 0x7fc01006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.380+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc020085ed0 con 0x7fc03006a950 2023-11-08T08:21:16.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.380+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0100682f0 0x7fc01006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.380+0000 7fc02f7fe700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0100682f0 0x7fc01006a7a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc018015040 tx=0x7fc01800a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.384+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc0200525b0 con 0x7fc03006a950 2023-11-08T08:21:16.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.566+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc03015e010 con 0x7fc03006a950 2023-11-08T08:21:16.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.567+0000 7fc027fff700 1 -- 172.21.15.116:0/1695094312 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+17909 (secure 0 0 0) 0x7fc03015e010 con 0x7fc03006a950 2023-11-08T08:21:16.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:16.569 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":44,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:34.014710+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:16.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0100682f0 msgr2=0x7fc01006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0100682f0 0x7fc01006a7a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc018015040 tx=0x7fc01800a000 comp rx=0 tx=0).stop 2023-11-08T08:21:16.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 msgr2=0x7fc0300c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 0x7fc0300c7030 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc020007290 tx=0x7fc020007650 comp rx=0 tx=0).stop 2023-11-08T08:21:16.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 shutdown_connections 2023-11-08T08:21:16.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0300c7570 0x7fc030150bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0100682f0 0x7fc01006a7a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01c003be0 0x7fc0300c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 --2- 172.21.15.116:0/1695094312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc03006a950 0x7fc0300c7030 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 >> 172.21.15.116:0/1695094312 conn(0x7fc03006c010 msgr2=0x7fc03006e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:16.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 shutdown_connections 2023-11-08T08:21:16.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.570+0000 7fc035b8e700 1 -- 172.21.15.116:0/1695094312 wait complete. 2023-11-08T08:21:16.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:20:30.822342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:20:31.988683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:21:16.584 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:21:16.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.706+0000 7f9aca2b8700 1 Processor -- start 2023-11-08T08:21:16.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.706+0000 7f9aca2b8700 1 -- start start 2023-11-08T08:21:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9aca2b8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac41599b0 0x7f9ac415d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9aca2b8700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac4159ec0 con 0x7f9ac40b1650 2023-11-08T08:21:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9aca2b8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac415aa60 con 0x7f9ac40af0a0 2023-11-08T08:21:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9aca2b8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac415d8a0 con 0x7f9ac41599b0 2023-11-08T08:21:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac3fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9ac40b1650 0x7f9ac40b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53262/0 (socket says 172.21.15.116:53262) 2023-11-08T08:21:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac3fff700 1 -- 172.21.15.116:0/2001971967 learned_addr learned my addr 172.21.15.116:0/2001971967 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881137183 0 0) 0x7f9ac4159ec0 con 0x7f9ac40b1650 2023-11-08T08:21:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac0035b0 con 0x7f9ac40b1650 2023-11-08T08:21:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac37fe700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac41599b0 0x7f9ac415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089482151 0 0) 0x7f9aac0035b0 con 0x7f9ac40b1650 2023-11-08T08:21:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac41599b0 msgr2=0x7f9ac415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac41599b0 0x7f9ac415d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 >> v1:172.21.15.129:6789/0 conn(0x7f9ac40af0a0 legacy=0x7f9ac415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac415db80 con 0x7f9ac40b1650 2023-11-08T08:21:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9ab4001670 con 0x7f9ac40b1650 2023-11-08T08:21:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.707+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 >> v1:172.21.15.116:6789/0 conn(0x7f9ac40b1650 legacy=0x7f9ac40b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9aac003a30 0x7f9aac003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac004850 0x7f9aac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9aac005200 0x7f9aac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac415d8a0 con 0x7f9aac003a30 2023-11-08T08:21:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac00c080 con 0x7f9aac004850 2023-11-08T08:21:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac00c240 con 0x7f9aac005200 2023-11-08T08:21:16.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac37fe700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac41599b0 0x7f9ac415d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:16.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9aac003a30 0x7f9aac003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac8855700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9aac005200 0x7f9aac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac37fe700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac004850 0x7f9aac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 -- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9aac005200 msgr2=0x7f9aac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9aac005200 0x7f9aac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 -- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac004850 msgr2=0x7f9aac004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac004850 0x7f9aac004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 -- 172.21.15.116:0/2001971967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0035b0 con 0x7f9aac003a30 2023-11-08T08:21:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac8855700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9aac005200 0x7f9aac00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9aac003a30 0x7f9aac003e40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c240 tx=0x7f9aac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c418a00bc9077f75 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4007c10 con 0x7f9aac003a30 2023-11-08T08:21:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab400ce00 con 0x7f9aac003a30 2023-11-08T08:21:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.708+0000 7f9ac2ffd700 1 -- 172.21.15.116:0/2001971967 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4005ba0 con 0x7f9aac003a30 2023-11-08T08:21:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9aca2b8700 1 -- 172.21.15.116:0/2001971967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9aac003a30 msgr2=0x7f9aac003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/2001971967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9aac003a30 0x7f9aac003e40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c240 tx=0x7f9aac005880 comp rx=0 tx=0).stop 2023-11-08T08:21:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9ac8855700 1 -- 172.21.15.116:0/2001971967 reap_dead start 2023-11-08T08:21:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9aca2b8700 1 -- 172.21.15.116:0/2001971967 shutdown_connections 2023-11-08T08:21:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9aca2b8700 1 -- 172.21.15.116:0/2001971967 >> 172.21.15.116:0/2001971967 conn(0x7f9ac406c010 msgr2=0x7f9ac40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9aca2b8700 1 -- 172.21.15.116:0/2001971967 shutdown_connections 2023-11-08T08:21:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.709+0000 7f9aca2b8700 1 -- 172.21.15.116:0/2001971967 wait complete. 2023-11-08T08:21:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.710+0000 7f9aca2b8700 1 Processor -- start 2023-11-08T08:21:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.710+0000 7f9aca2b8700 1 -- start start 2023-11-08T08:21:16.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac003a30 0x7f9ac41f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 0x7f9ac4158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac40c3700 0x7f9ac40c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41e2e50 con 0x7f9ac40cb930 2023-11-08T08:21:16.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41e30a0 con 0x7f9aac003a30 2023-11-08T08:21:16.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41e3380 con 0x7f9ac40c3700 2023-11-08T08:21:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 0x7f9ac4158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 0x7f9ac4158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53822/0 (socket says 172.21.15.116:53822) 2023-11-08T08:21:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 -- 172.21.15.116:0/804784087 learned_addr learned my addr 172.21.15.116:0/804784087 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac003a30 0x7f9ac41f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac8855700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac40c3700 0x7f9ac40c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 -- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac40c3700 msgr2=0x7f9ac40c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac40c3700 0x7f9ac40c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 -- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac003a30 msgr2=0x7f9ac41f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac003a30 0x7f9ac41f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 -- 172.21.15.116:0/804784087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac415d050 con 0x7f9ac40cb930 2023-11-08T08:21:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac37fe700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 0x7f9ac4158dd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80126e0 tx=0x7f9ab80129f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8004220 con 0x7f9ac40cb930 2023-11-08T08:21:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab800c070 con 0x7f9ac40cb930 2023-11-08T08:21:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.711+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac415de60 con 0x7f9ac40cb930 2023-11-08T08:21:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.712+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80036a0 con 0x7f9ac40cb930 2023-11-08T08:21:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.712+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac415d1e0 con 0x7f9ac40cb930 2023-11-08T08:21:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.712+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac40743f0 con 0x7f9ac40cb930 2023-11-08T08:21:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.713+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ab8003800 con 0x7f9ac40cb930 2023-11-08T08:21:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.713+0000 7f9ac0ff9700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9ab00682f0 0x7f9ab006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.713+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9ab8087660 con 0x7f9ac40cb930 2023-11-08T08:21:16.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.713+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9ab00682f0 0x7f9ab006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:16.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.714+0000 7f9ac3fff700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9ab00682f0 0x7f9ab006a7a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4001930 tx=0x7f9ab4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:16.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.717+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ab80525e0 con 0x7f9ac40cb930 2023-11-08T08:21:16.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.896+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9ac406e620 con 0x7f9ac40cb930 2023-11-08T08:21:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.896+0000 7f9ac0ff9700 1 -- 172.21.15.116:0/804784087 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9ab8053ec0 con 0x7f9ac40cb930 2023-11-08T08:21:16.898 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:21:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9ab00682f0 msgr2=0x7f9ab006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9ab00682f0 0x7f9ab006a7a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4001930 tx=0x7f9ab4007000 comp rx=0 tx=0).stop 2023-11-08T08:21:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 msgr2=0x7f9ac4158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 0x7f9ac4158dd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80126e0 tx=0x7f9ab80129f0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 shutdown_connections 2023-11-08T08:21:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ac40c3700 0x7f9ac40c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9ab00682f0 0x7f9ab006a7a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac003a30 0x7f9ac41f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 --2- 172.21.15.116:0/804784087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ac40cb930 0x7f9ac4158dd0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 >> 172.21.15.116:0/804784087 conn(0x7f9ac406c010 msgr2=0x7f9ac406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 shutdown_connections 2023-11-08T08:21:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:16.898+0000 7f9aca2b8700 1 -- 172.21.15.116:0/804784087 wait complete. 2023-11-08T08:21:16.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:21:17.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.032+0000 7f95e0512700 1 Processor -- start 2023-11-08T08:21:17.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.032+0000 7f95e0512700 1 -- start start 2023-11-08T08:21:17.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.033+0000 7f95e0512700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d806a980 0x7f95d80cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.033+0000 7f95e0512700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95d80cdd50 0x7f95d80cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.033+0000 7f95e0512700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d815e0d0 con 0x7f95d815a4d0 2023-11-08T08:21:17.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.033+0000 7f95e0512700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d80ce560 con 0x7f95d80cdd50 2023-11-08T08:21:17.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.033+0000 7f95e0512700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d815e320 con 0x7f95d806a980 2023-11-08T08:21:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95d80cdd50 0x7f95d80cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95d80cdd50 0x7f95d80cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53824/0 (socket says 172.21.15.116:53824) 2023-11-08T08:21:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 -- 172.21.15.116:0/418287799 learned_addr learned my addr 172.21.15.116:0/418287799 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95de2ae700 1 --1- 172.21.15.116:0/418287799 >> v1:172.21.15.116:6789/0 conn(0x7f95d815a4d0 0x7f95d806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53272/0 (socket says 172.21.15.116:53272) 2023-11-08T08:21:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 -- 172.21.15.116:0/418287799 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d806a980 msgr2=0x7f95d80cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 --2- 172.21.15.116:0/418287799 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d806a980 0x7f95d80cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 -- 172.21.15.116:0/418287799 >> v1:172.21.15.116:6789/0 conn(0x7f95d815a4d0 legacy=0x7f95d806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 -- 172.21.15.116:0/418287799 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d815e600 con 0x7f95d80cdd50 2023-11-08T08:21:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95dd2ac700 1 -- 172.21.15.116:0/418287799 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867278437 0 0) 0x7f95d815e0d0 con 0x7f95d815a4d0 2023-11-08T08:21:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95deaaf700 1 --2- 172.21.15.116:0/418287799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d80cdd50 0x7f95d80cb010 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009e50 tx=0x7f95d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c219c979cd90c135 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95dd2ac700 1 -- 172.21.15.116:0/418287799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d000e070 con 0x7f95d80cdd50 2023-11-08T08:21:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95dd2ac700 1 -- 172.21.15.116:0/418287799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d000b9c0 con 0x7f95d80cdd50 2023-11-08T08:21:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.034+0000 7f95dd2ac700 1 -- 172.21.15.116:0/418287799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d000bd70 con 0x7f95d80cdd50 2023-11-08T08:21:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 -- 172.21.15.116:0/418287799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d80cdd50 msgr2=0x7f95d80cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 --2- 172.21.15.116:0/418287799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d80cdd50 0x7f95d80cb010 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009e50 tx=0x7f95d000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:17.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 -- 172.21.15.116:0/418287799 shutdown_connections 2023-11-08T08:21:17.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 --2- 172.21.15.116:0/418287799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d80cdd50 0x7f95d80cb010 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 --2- 172.21.15.116:0/418287799 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d806a980 0x7f95d80cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 -- 172.21.15.116:0/418287799 >> 172.21.15.116:0/418287799 conn(0x7f95d806c010 msgr2=0x7f95d80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:17.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 -- 172.21.15.116:0/418287799 shutdown_connections 2023-11-08T08:21:17.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.035+0000 7f95e0512700 1 -- 172.21.15.116:0/418287799 wait complete. 2023-11-08T08:21:17.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.036+0000 7f95e0512700 1 Processor -- start 2023-11-08T08:21:17.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.036+0000 7f95e0512700 1 -- start start 2023-11-08T08:21:17.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d806a980 0x7f95d80c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d80cdd50 0x7f95d8153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 0x7f95d8159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d8154330 con 0x7f95d815a4d0 2023-11-08T08:21:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80c8370 con 0x7f95d806a980 2023-11-08T08:21:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80c8650 con 0x7f95d80cdd50 2023-11-08T08:21:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 0x7f95d8159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 0x7f95d8159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53834/0 (socket says 172.21.15.116:53834) 2023-11-08T08:21:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 -- 172.21.15.116:0/268302658 learned_addr learned my addr 172.21.15.116:0/268302658 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95de2ae700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d806a980 0x7f95d80c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95ddaad700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d80cdd50 0x7f95d8153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 -- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d80cdd50 msgr2=0x7f95d8153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d80cdd50 0x7f95d8153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 -- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d806a980 msgr2=0x7f95d80c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d806a980 0x7f95d80c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 -- 172.21.15.116:0/268302658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d0009b30 con 0x7f95d815a4d0 2023-11-08T08:21:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95ddaad700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d80cdd50 0x7f95d8153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95deaaf700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 0x7f95d8159100 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f95d000b730 tx=0x7f95d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004960 con 0x7f95d815a4d0 2023-11-08T08:21:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d00099b0 con 0x7f95d815a4d0 2023-11-08T08:21:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d815a8a0 con 0x7f95d815a4d0 2023-11-08T08:21:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d001c680 con 0x7f95d815a4d0 2023-11-08T08:21:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.037+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d815e8e0 con 0x7f95d815a4d0 2023-11-08T08:21:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.038+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d815aa30 con 0x7f95d815a4d0 2023-11-08T08:21:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.039+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f95d0022030 con 0x7f95d815a4d0 2023-11-08T08:21:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.039+0000 7f95ceffd700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f95bc0682f0 0x7f95bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.039+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f95d0087280 con 0x7f95d815a4d0 2023-11-08T08:21:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.039+0000 7f95de2ae700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f95bc0682f0 0x7f95bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.039+0000 7f95de2ae700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f95bc0682f0 0x7f95bc06a7a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f95c8000f60 tx=0x7f95c8007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.043+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95d815aa30 con 0x7f95d815a4d0 2023-11-08T08:21:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.222+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95d815aa30 con 0x7f95d815a4d0 2023-11-08T08:21:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.222+0000 7f95ceffd700 1 -- 172.21.15.116:0/268302658 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95d815aa30 con 0x7f95d815a4d0 2023-11-08T08:21:17.224 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:21:17.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f95bc0682f0 msgr2=0x7f95bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f95bc0682f0 0x7f95bc06a7a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f95c8000f60 tx=0x7f95c8007e70 comp rx=0 tx=0).stop 2023-11-08T08:21:17.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 msgr2=0x7f95d8159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 0x7f95d8159100 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f95d000b730 tx=0x7f95d0004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 shutdown_connections 2023-11-08T08:21:17.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d815a4d0 0x7f95d8159100 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f95bc0682f0 0x7f95bc06a7a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d806a980 0x7f95d80c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 --2- 172.21.15.116:0/268302658 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f95d80cdd50 0x7f95d8153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.224+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 >> 172.21.15.116:0/268302658 conn(0x7f95d806c010 msgr2=0x7f95d8154af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.225+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 shutdown_connections 2023-11-08T08:21:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.225+0000 7f95e0512700 1 -- 172.21.15.116:0/268302658 wait complete. 2023-11-08T08:21:17.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:21:17.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.359+0000 7f8e0e957700 1 Processor -- start 2023-11-08T08:21:17.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.359+0000 7f8e0e957700 1 -- start start 2023-11-08T08:21:17.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e0e957700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e080718e0 0x7f8e0806c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e0e957700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e08160a50 con 0x7f8e0815c810 2023-11-08T08:21:17.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e0e957700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e08160d30 con 0x7f8e0806cdc0 2023-11-08T08:21:17.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e0e957700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0815cbe0 con 0x7f8e080718e0 2023-11-08T08:21:17.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e077fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e080718e0 0x7f8e0806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e07fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8e0815c810 0x7f8e08160260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53280/0 (socket says 172.21.15.116:53280) 2023-11-08T08:21:17.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e07fff700 1 -- 172.21.15.116:0/555856298 learned_addr learned my addr 172.21.15.116:0/555856298 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:17.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e077fe700 1 -- 172.21.15.116:0/555856298 >> v1:172.21.15.129:6789/0 conn(0x7f8e0806cdc0 legacy=0x7f8e0806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e077fe700 1 -- 172.21.15.116:0/555856298 >> v1:172.21.15.116:6789/0 conn(0x7f8e0815c810 legacy=0x7f8e08160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e06ffd700 1 -- 172.21.15.116:0/555856298 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504712579 0 0) 0x7f8e08160a50 con 0x7f8e0815c810 2023-11-08T08:21:17.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.360+0000 7f8e077fe700 1 -- 172.21.15.116:0/555856298 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e08161010 con 0x7f8e080718e0 2023-11-08T08:21:17.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e077fe700 1 --2- 172.21.15.116:0/555856298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e080718e0 0x7f8e0806c880 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009870 tx=0x7f8df8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948d70b251904f05 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e06ffd700 1 -- 172.21.15.116:0/555856298 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8011830 con 0x7f8e080718e0 2023-11-08T08:21:17.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e06ffd700 1 -- 172.21.15.116:0/555856298 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df8011990 con 0x7f8e080718e0 2023-11-08T08:21:17.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e06ffd700 1 -- 172.21.15.116:0/555856298 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8011d20 con 0x7f8e080718e0 2023-11-08T08:21:17.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 -- 172.21.15.116:0/555856298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e080718e0 msgr2=0x7f8e0806c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 --2- 172.21.15.116:0/555856298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e080718e0 0x7f8e0806c880 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009870 tx=0x7f8df8009b80 comp rx=0 tx=0).stop 2023-11-08T08:21:17.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 -- 172.21.15.116:0/555856298 shutdown_connections 2023-11-08T08:21:17.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 --2- 172.21.15.116:0/555856298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e080718e0 0x7f8e0806c880 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 -- 172.21.15.116:0/555856298 >> 172.21.15.116:0/555856298 conn(0x7f8e0806c010 msgr2=0x7f8e080cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:17.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 -- 172.21.15.116:0/555856298 shutdown_connections 2023-11-08T08:21:17.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.361+0000 7f8e0e957700 1 -- 172.21.15.116:0/555856298 wait complete. 2023-11-08T08:21:17.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.363+0000 7f8e0e957700 1 Processor -- start 2023-11-08T08:21:17.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.363+0000 7f8e0e957700 1 -- start start 2023-11-08T08:21:17.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0e957700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e0806cdc0 0x7f8e0814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0e957700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8e080718e0 0x7f8e080af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0e957700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 0x7f8e080b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0e957700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b58f0 con 0x7f8e0815c810 2023-11-08T08:21:17.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0e957700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0814c5e0 con 0x7f8e0806cdc0 2023-11-08T08:21:17.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0e957700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080af9e0 con 0x7f8e080718e0 2023-11-08T08:21:17.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 0x7f8e080b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 0x7f8e080b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53844/0 (socket says 172.21.15.116:53844) 2023-11-08T08:21:17.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 -- 172.21.15.116:0/1359824786 learned_addr learned my addr 172.21.15.116:0/1359824786 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:17.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e07fff700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e0806cdc0 0x7f8e0814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e077fe700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8e080718e0 0x7f8e080af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 -- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8e080718e0 msgr2=0x7f8e080af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8e080718e0 0x7f8e080af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 -- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e0806cdc0 msgr2=0x7f8e0814c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e0806cdc0 0x7f8e0814c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e0cef4700 1 -- 172.21.15.116:0/1359824786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80094a0 con 0x7f8e0815c810 2023-11-08T08:21:17.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.364+0000 7f8e07fff700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e0806cdc0 0x7f8e0814c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:17.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.365+0000 7f8e0cef4700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 0x7f8e080b3ff0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000cbc0 tx=0x7f8e0000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.365+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00003c00 con 0x7f8e0815c810 2023-11-08T08:21:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.365+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00003d60 con 0x7f8e0815c810 2023-11-08T08:21:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.365+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e080ce740 con 0x7f8e0815c810 2023-11-08T08:21:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.365+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000abf0 con 0x7f8e0815c810 2023-11-08T08:21:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.365+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e080ceb20 con 0x7f8e0815c810 2023-11-08T08:21:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.366+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e080ce8d0 con 0x7f8e0815c810 2023-11-08T08:21:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.366+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e00010050 con 0x7f8e0815c810 2023-11-08T08:21:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.366+0000 7f8e04ff9700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8df40682f0 0x7f8df406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.367+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8e00085c30 con 0x7f8e0815c810 2023-11-08T08:21:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.367+0000 7f8e07fff700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8df40682f0 0x7f8df406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.367+0000 7f8e07fff700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8df40682f0 0x7f8df406a7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc000f60 tx=0x7f8dfc007e50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.371+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e00052490 con 0x7f8e0815c810 2023-11-08T08:21:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.552+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e080d0640 con 0x7f8e0815c810 2023-11-08T08:21:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.552+0000 7f8e04ff9700 1 -- 172.21.15.116:0/1359824786 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e00055ab0 con 0x7f8e0815c810 2023-11-08T08:21:17.554 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:21:17.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8df40682f0 msgr2=0x7f8df406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8df40682f0 0x7f8df406a7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc000f60 tx=0x7f8dfc007e50 comp rx=0 tx=0).stop 2023-11-08T08:21:17.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 msgr2=0x7f8e080b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 0x7f8e080b3ff0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000cbc0 tx=0x7f8e0000cf80 comp rx=0 tx=0).stop 2023-11-08T08:21:17.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 shutdown_connections 2023-11-08T08:21:17.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0815c810 0x7f8e080b3ff0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8df40682f0 0x7f8df406a7a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e0806cdc0 0x7f8e0814c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 --2- 172.21.15.116:0/1359824786 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8e080718e0 0x7f8e080af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 >> 172.21.15.116:0/1359824786 conn(0x7f8e0806c010 msgr2=0x7f8e080adf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:17.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 shutdown_connections 2023-11-08T08:21:17.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.554+0000 7f8e0e957700 1 -- 172.21.15.116:0/1359824786 wait complete. 2023-11-08T08:21:17.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:21:17.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.689+0000 7fb31a0f1700 1 Processor -- start 2023-11-08T08:21:17.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.689+0000 7fb31a0f1700 1 -- start start 2023-11-08T08:21:17.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb31a0f1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb31415a4d0 0x7fb31406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb31a0f1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31415e0d0 con 0x7fb3140cdf20 2023-11-08T08:21:17.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb31a0f1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140ce310 con 0x7fb31406a7b0 2023-11-08T08:21:17.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb31a0f1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31406ac30 con 0x7fb31415a4d0 2023-11-08T08:21:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb313fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb3140cdf20 0x7fb3140cb010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53292/0 (socket says 172.21.15.116:53292) 2023-11-08T08:21:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb313fff700 1 -- 172.21.15.116:0/14982859 learned_addr learned my addr 172.21.15.116:0/14982859 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3137fe700 1 --2- 172.21.15.116:0/14982859 >> v2:172.21.15.129:3300/0 conn(0x7fb31415a4d0 0x7fb31406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549854888 0 0) 0x7fb31415e0d0 con 0x7fb3140cdf20 2023-11-08T08:21:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc0035b0 con 0x7fb3140cdf20 2023-11-08T08:21:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707251682 0 0) 0x7fb3140ce310 con 0x7fb31406a7b0 2023-11-08T08:21:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31415e0d0 con 0x7fb31406a7b0 2023-11-08T08:21:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3137fe700 1 -- 172.21.15.116:0/14982859 >> v1:172.21.15.129:6789/0 conn(0x7fb31406a7b0 legacy=0x7fb3140cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3137fe700 1 -- 172.21.15.116:0/14982859 >> v1:172.21.15.116:6789/0 conn(0x7fb3140cdf20 legacy=0x7fb3140cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3137fe700 1 -- 172.21.15.116:0/14982859 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140ce560 con 0x7fb31415a4d0 2023-11-08T08:21:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.690+0000 7fb3137fe700 1 --2- 172.21.15.116:0/14982859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31415a4d0 0x7fb31406a270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb308009840 tx=0x7fb308009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df970a226bdc0b4f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308011830 con 0x7fb31415a4d0 2023-11-08T08:21:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308011990 con 0x7fb31415a4d0 2023-11-08T08:21:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb3127fc700 1 -- 172.21.15.116:0/14982859 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308011d20 con 0x7fb31415a4d0 2023-11-08T08:21:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 -- 172.21.15.116:0/14982859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31415a4d0 msgr2=0x7fb31406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/14982859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31415a4d0 0x7fb31406a270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb308009840 tx=0x7fb308009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 -- 172.21.15.116:0/14982859 shutdown_connections 2023-11-08T08:21:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/14982859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31415a4d0 0x7fb31406a270 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 -- 172.21.15.116:0/14982859 >> 172.21.15.116:0/14982859 conn(0x7fb31406c010 msgr2=0x7fb3140cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 -- 172.21.15.116:0/14982859 shutdown_connections 2023-11-08T08:21:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.691+0000 7fb31a0f1700 1 -- 172.21.15.116:0/14982859 wait complete. 2023-11-08T08:21:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.692+0000 7fb31a0f1700 1 Processor -- start 2023-11-08T08:21:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 -- start start 2023-11-08T08:21:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31406a7b0 0x7fb3140c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3140cdf20 0x7fb31415c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 0x7fb3140aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415cf10 con 0x7fb31415a4d0 2023-11-08T08:21:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415d1f0 con 0x7fb31406a7b0 2023-11-08T08:21:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb31a0f1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415d4d0 con 0x7fb3140cdf20 2023-11-08T08:21:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb313fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 0x7fb3140aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb313fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 0x7fb3140aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53858/0 (socket says 172.21.15.116:53858) 2023-11-08T08:21:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.693+0000 7fb313fff700 1 -- 172.21.15.116:0/433039764 learned_addr learned my addr 172.21.15.116:0/433039764 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb313fff700 1 -- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3140cdf20 msgr2=0x7fb31415c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb313fff700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3140cdf20 0x7fb31415c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb3137fe700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31406a7b0 0x7fb3140c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb313fff700 1 -- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31406a7b0 msgr2=0x7fb3140c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb313fff700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31406a7b0 0x7fb3140c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb313fff700 1 -- 172.21.15.116:0/433039764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080094a0 con 0x7fb31415a4d0 2023-11-08T08:21:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb313fff700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 0x7fb3140aeca0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00cbc0 tx=0x7fb30c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c00ab50 con 0x7fb31415a4d0 2023-11-08T08:21:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c00acb0 con 0x7fb31415a4d0 2023-11-08T08:21:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31415e600 con 0x7fb31415a4d0 2023-11-08T08:21:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c012600 con 0x7fb31415a4d0 2023-11-08T08:21:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.694+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31415e320 con 0x7fb31415a4d0 2023-11-08T08:21:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.695+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31415e8e0 con 0x7fb31415a4d0 2023-11-08T08:21:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.695+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb30c019030 con 0x7fb31415a4d0 2023-11-08T08:21:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.696+0000 7fb2fbfff700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3000682f0 0x7fb30006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.696+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb30c0858f0 con 0x7fb31415a4d0 2023-11-08T08:21:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.696+0000 7fb3137fe700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3000682f0 0x7fb30006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.696+0000 7fb3137fe700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3000682f0 0x7fb30006a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb308009b90 tx=0x7fb30801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.700+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb30c04b5e0 con 0x7fb31415a4d0 2023-11-08T08:21:17.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.880+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3140743f0 con 0x7fb31415a4d0 2023-11-08T08:21:17.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.880+0000 7fb2fbfff700 1 -- 172.21.15.116:0/433039764 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb30c017020 con 0x7fb31415a4d0 2023-11-08T08:21:17.882 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:21:17.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.882+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3000682f0 msgr2=0x7fb30006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.882+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3000682f0 0x7fb30006a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb308009b90 tx=0x7fb30801c040 comp rx=0 tx=0).stop 2023-11-08T08:21:17.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 msgr2=0x7fb3140aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:17.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 0x7fb3140aeca0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00cbc0 tx=0x7fb30c00cf80 comp rx=0 tx=0).stop 2023-11-08T08:21:17.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 shutdown_connections 2023-11-08T08:21:17.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31415a4d0 0x7fb3140aeca0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb3000682f0 0x7fb30006a7a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb31406a7b0 0x7fb3140c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 --2- 172.21.15.116:0/433039764 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb3140cdf20 0x7fb31415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:17.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 >> 172.21.15.116:0/433039764 conn(0x7fb31406c010 msgr2=0x7fb3141548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:17.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 shutdown_connections 2023-11-08T08:21:17.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:17.883+0000 7fb31a0f1700 1 -- 172.21.15.116:0/433039764 wait complete. 2023-11-08T08:21:17.896 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:18.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.014+0000 7f1f7374c700 1 Processor -- start 2023-11-08T08:21:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.014+0000 7f1f7374c700 1 -- start start 2023-11-08T08:21:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f7374c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 0x7f1f6c06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f7374c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6c06ab80 con 0x7f1f6c06e570 2023-11-08T08:21:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f7374c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6c0b1810 con 0x7f1f6c15cc80 2023-11-08T08:21:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f7374c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c0b2490 con 0x7f1f6c06a270 2023-11-08T08:21:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f71ce9700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1f6c06e570 0x7f1f6c070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53306/0 (socket says 172.21.15.116:53306) 2023-11-08T08:21:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f71ce9700 1 -- 172.21.15.116:0/2659717687 learned_addr learned my addr 172.21.15.116:0/2659717687 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788956325 0 0) 0x7f1f6c06ab80 con 0x7f1f6c06e570 2023-11-08T08:21:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f580035b0 con 0x7f1f6c06e570 2023-11-08T08:21:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.015+0000 7f1f714e8700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 0x7f1f6c06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190996228 0 0) 0x7f1f580035b0 con 0x7f1f6c06e570 2023-11-08T08:21:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 msgr2=0x7f1f6c06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 0x7f1f6c06a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 >> v1:172.21.15.129:6789/0 conn(0x7f1f6c15cc80 legacy=0x7f1f6c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c0b1af0 con 0x7f1f6c06e570 2023-11-08T08:21:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f1f640019f0 con 0x7f1f6c06e570 2023-11-08T08:21:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 >> v1:172.21.15.116:6789/0 conn(0x7f1f6c06e570 legacy=0x7f1f6c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f58003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f58004850 0x7f1f58004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 0x7f1f5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c0b2490 con 0x7f1f58003a30 2023-11-08T08:21:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58005580 con 0x7f1f58004850 2023-11-08T08:21:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5800bf50 con 0x7f1f6c06a270 2023-11-08T08:21:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f58003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f70ce7700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f58004850 0x7f1f58004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f71ce9700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 0x7f1f5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 -- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 msgr2=0x7f1f5800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c06a270 0x7f1f5800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 -- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f58004850 msgr2=0x7f1f58004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f58004850 0x7f1f58004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 -- 172.21.15.116:0/2659717687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580035b0 con 0x7f1f58003a30 2023-11-08T08:21:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f714e8700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f58003e40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009890 tx=0x7f1f58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ea292e4ac5152f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c00c070 con 0x7f1f58003a30 2023-11-08T08:21:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5c013610 con 0x7f1f58003a30 2023-11-08T08:21:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.016+0000 7f1f63fff700 1 -- 172.21.15.116:0/2659717687 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0139a0 con 0x7f1f58003a30 2023-11-08T08:21:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f7374c700 1 -- 172.21.15.116:0/2659717687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 msgr2=0x7f1f58003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f7374c700 1 --2- 172.21.15.116:0/2659717687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f58003e40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009890 tx=0x7f1f58005880 comp rx=0 tx=0).stop 2023-11-08T08:21:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f71ce9700 1 -- 172.21.15.116:0/2659717687 reap_dead start 2023-11-08T08:21:18.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f7374c700 1 -- 172.21.15.116:0/2659717687 shutdown_connections 2023-11-08T08:21:18.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f7374c700 1 -- 172.21.15.116:0/2659717687 >> 172.21.15.116:0/2659717687 conn(0x7f1f6c06c010 msgr2=0x7f1f6c0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:18.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f7374c700 1 -- 172.21.15.116:0/2659717687 shutdown_connections 2023-11-08T08:21:18.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.017+0000 7f1f7374c700 1 -- 172.21.15.116:0/2659717687 wait complete. 2023-11-08T08:21:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.018+0000 7f1f7374c700 1 Processor -- start 2023-11-08T08:21:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.018+0000 7f1f7374c700 1 -- start start 2023-11-08T08:21:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f7374c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f6c153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f7374c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c1505d0 0x7f1f6c0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f7374c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f6c1554d0 0x7f1f6c0c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f7374c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c06e9b0 con 0x7f1f58003a30 2023-11-08T08:21:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f7374c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c1550b0 con 0x7f1f6c1554d0 2023-11-08T08:21:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f7374c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c154430 con 0x7f1f6c1505d0 2023-11-08T08:21:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f6c153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f6c153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53872/0 (socket says 172.21.15.116:53872) 2023-11-08T08:21:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 -- 172.21.15.116:0/1693550619 learned_addr learned my addr 172.21.15.116:0/1693550619 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f70ce7700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c1505d0 0x7f1f6c0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f71ce9700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f6c1554d0 0x7f1f6c0c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 -- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c1505d0 msgr2=0x7f1f6c0b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c1505d0 0x7f1f6c0b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 -- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f6c1554d0 msgr2=0x7f1f6c0c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f6c1554d0 0x7f1f6c0c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 -- 172.21.15.116:0/1693550619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c0b20b0 con 0x7f1f58003a30 2023-11-08T08:21:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.019+0000 7f1f714e8700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f6c153da0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c016040 tx=0x7f1f5c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.020+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c013a50 con 0x7f1f58003a30 2023-11-08T08:21:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.020+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5c013bd0 con 0x7f1f58003a30 2023-11-08T08:21:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.020+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c0b1dd0 con 0x7f1f58003a30 2023-11-08T08:21:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.020+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0036a0 con 0x7f1f58003a30 2023-11-08T08:21:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.020+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c15daa0 con 0x7f1f58003a30 2023-11-08T08:21:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.021+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c0743f0 con 0x7f1f58003a30 2023-11-08T08:21:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.022+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f5c02c030 con 0x7f1f58003a30 2023-11-08T08:21:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.022+0000 7f1f61ffb700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f500682f0 0x7f1f5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.022+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1f5c086830 con 0x7f1f58003a30 2023-11-08T08:21:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.022+0000 7f1f70ce7700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f500682f0 0x7f1f5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.022+0000 7f1f70ce7700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f500682f0 0x7f1f5006a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1f68007fd0 tx=0x7f1f68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.026+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f5c053130 con 0x7f1f58003a30 2023-11-08T08:21:18.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.247+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f6c0cd5f0 con 0x7f1f58003a30 2023-11-08T08:21:18.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.247+0000 7f1f61ffb700 1 -- 172.21.15.116:0/1693550619 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f1f5c056750 con 0x7f1f58003a30 2023-11-08T08:21:18.249 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:18.249 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:21:18.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f500682f0 msgr2=0x7f1f5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f500682f0 0x7f1f5006a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1f68007fd0 tx=0x7f1f68009040 comp rx=0 tx=0).stop 2023-11-08T08:21:18.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 msgr2=0x7f1f6c153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f6c153da0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c016040 tx=0x7f1f5c0047c0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 shutdown_connections 2023-11-08T08:21:18.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1f500682f0 0x7f1f5006a7a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f6c1554d0 0x7f1f6c0c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58003a30 0x7f1f6c153da0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 --2- 172.21.15.116:0/1693550619 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f6c1505d0 0x7f1f6c0b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 >> 172.21.15.116:0/1693550619 conn(0x7f1f6c06c010 msgr2=0x7f1f6c0cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 shutdown_connections 2023-11-08T08:21:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.249+0000 7f1f7374c700 1 -- 172.21.15.116:0/1693550619 wait complete. 2023-11-08T08:21:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:21:18.262 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:18.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.380+0000 7f0848bb3700 1 Processor -- start 2023-11-08T08:21:18.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.380+0000 7f0848bb3700 1 -- start start 2023-11-08T08:21:18.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.381+0000 7f0848bb3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 0x7f084406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.381+0000 7f0848bb3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084406ab80 con 0x7f084406e570 2023-11-08T08:21:18.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.381+0000 7f0848bb3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08440b1810 con 0x7f084415cc80 2023-11-08T08:21:18.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.381+0000 7f0848bb3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08440b2490 con 0x7f084406a270 2023-11-08T08:21:18.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0842d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f084406e570 0x7f0844070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53318/0 (socket says 172.21.15.116:53318) 2023-11-08T08:21:18.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0842d9d700 1 -- 172.21.15.116:0/2399889873 learned_addr learned my addr 172.21.15.116:0/2399889873 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:18.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084259c700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 0x7f084406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321755616 0 0) 0x7f084406ab80 con 0x7f084406e570 2023-11-08T08:21:18.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f082c0035b0 con 0x7f084406e570 2023-11-08T08:21:18.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731692740 0 0) 0x7f082c0035b0 con 0x7f084406e570 2023-11-08T08:21:18.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 msgr2=0x7f084406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 0x7f084406a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 >> v1:172.21.15.129:6789/0 conn(0x7f084415cc80 legacy=0x7f084406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440b1af0 con 0x7f084406e570 2023-11-08T08:21:18.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f083c0019f0 con 0x7f084406e570 2023-11-08T08:21:18.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 >> v1:172.21.15.116:6789/0 conn(0x7f084406e570 legacy=0x7f0844070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c003a30 0x7f082c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f082c004850 0x7f082c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 0x7f082c005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440b2490 con 0x7f082c004850 2023-11-08T08:21:18.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c00bb40 con 0x7f082c003a30 2023-11-08T08:21:18.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c00bd00 con 0x7f084406a270 2023-11-08T08:21:18.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f082c004850 0x7f082c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f084259c700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c003a30 0x7f082c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0842d9d700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 0x7f082c005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0841d9b700 1 -- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 msgr2=0x7f082c005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f084406a270 0x7f082c005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0841d9b700 1 -- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c003a30 msgr2=0x7f082c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.382+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c003a30 0x7f082c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0841d9b700 1 -- 172.21.15.116:0/2399889873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0035b0 con 0x7f082c004850 2023-11-08T08:21:18.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f082c004850 0x7f082c004cc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f08380018d0 tx=0x7f082c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8beee457b223177 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838012620 con 0x7f082c004850 2023-11-08T08:21:18.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838012780 con 0x7f082c004850 2023-11-08T08:21:18.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f084159a700 1 -- 172.21.15.116:0/2399889873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838012b30 con 0x7f082c004850 2023-11-08T08:21:18.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0848bb3700 1 -- 172.21.15.116:0/2399889873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f082c004850 msgr2=0x7f082c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2399889873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f082c004850 0x7f082c004cc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f08380018d0 tx=0x7f082c005880 comp rx=0 tx=0).stop 2023-11-08T08:21:18.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0842d9d700 1 -- 172.21.15.116:0/2399889873 reap_dead start 2023-11-08T08:21:18.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0848bb3700 1 -- 172.21.15.116:0/2399889873 shutdown_connections 2023-11-08T08:21:18.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0848bb3700 1 -- 172.21.15.116:0/2399889873 >> 172.21.15.116:0/2399889873 conn(0x7f084406c010 msgr2=0x7f08440cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:18.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0848bb3700 1 -- 172.21.15.116:0/2399889873 shutdown_connections 2023-11-08T08:21:18.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.383+0000 7f0848bb3700 1 -- 172.21.15.116:0/2399889873 wait complete. 2023-11-08T08:21:18.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 Processor -- start 2023-11-08T08:21:18.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 -- start start 2023-11-08T08:21:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c004850 0x7f0844151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 0x7f08440b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08441e60b0 0x7f08441e6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084406e9b0 con 0x7f0844150b00 2023-11-08T08:21:18.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440b58e0 con 0x7f082c004850 2023-11-08T08:21:18.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.385+0000 7f0848bb3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0844150f70 con 0x7f08441e60b0 2023-11-08T08:21:18.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 0x7f08440b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 0x7f08440b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53882/0 (socket says 172.21.15.116:53882) 2023-11-08T08:21:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 -- 172.21.15.116:0/2453044518 learned_addr learned my addr 172.21.15.116:0/2453044518 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f084259c700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c004850 0x7f0844151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0842d9d700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08441e60b0 0x7f08441e6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 -- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08441e60b0 msgr2=0x7f08441e6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08441e60b0 0x7f08441e6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 -- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c004850 msgr2=0x7f0844151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c004850 0x7f0844151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 -- 172.21.15.116:0/2453044518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440b20b0 con 0x7f0844150b00 2023-11-08T08:21:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0842d9d700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08441e60b0 0x7f08441e6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:18.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0841d9b700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 0x7f08440b06a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f083800c0e0 tx=0x7f0838001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838012af0 con 0x7f0844150b00 2023-11-08T08:21:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838021070 con 0x7f0844150b00 2023-11-08T08:21:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838012e90 con 0x7f0844150b00 2023-11-08T08:21:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08440b1dd0 con 0x7f0844150b00 2023-11-08T08:21:18.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.386+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084415daa0 con 0x7f0844150b00 2023-11-08T08:21:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.387+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08440743f0 con 0x7f0844150b00 2023-11-08T08:21:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.388+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0838010050 con 0x7f0844150b00 2023-11-08T08:21:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.388+0000 7f0832ffd700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08240682f0 0x7f082406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.388+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0838086f30 con 0x7f0844150b00 2023-11-08T08:21:18.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.388+0000 7f084259c700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08240682f0 0x7f082406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.388+0000 7f084259c700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08240682f0 0x7f082406a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f084415e5d0 tx=0x7f0834009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.392+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08380536a0 con 0x7f0844150b00 2023-11-08T08:21:18.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.574+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08440cd5f0 con 0x7f0844150b00 2023-11-08T08:21:18.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.575+0000 7f0832ffd700 1 -- 172.21.15.116:0/2453044518 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+17909 (secure 0 0 0) 0x7f0838056cc0 con 0x7f0844150b00 2023-11-08T08:21:18.576 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:18.576 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":44,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:34.014710+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:18.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08240682f0 msgr2=0x7f082406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08240682f0 0x7f082406a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f084415e5d0 tx=0x7f0834009210 comp rx=0 tx=0).stop 2023-11-08T08:21:18.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 msgr2=0x7f08440b06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 0x7f08440b06a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f083800c0e0 tx=0x7f0838001230 comp rx=0 tx=0).stop 2023-11-08T08:21:18.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 shutdown_connections 2023-11-08T08:21:18.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f08441e60b0 0x7f08441e6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f08240682f0 0x7f082406a7a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f082c004850 0x7f0844151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 --2- 172.21.15.116:0/2453044518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0844150b00 0x7f08440b06a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 >> 172.21.15.116:0/2453044518 conn(0x7f084406c010 msgr2=0x7f08441523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:18.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 shutdown_connections 2023-11-08T08:21:18.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.577+0000 7f0848bb3700 1 -- 172.21.15.116:0/2453044518 wait complete. 2023-11-08T08:21:18.591 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-08T08:21:18.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.710+0000 7f9f9e0d4700 1 Processor -- start 2023-11-08T08:21:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.710+0000 7f9f9e0d4700 1 -- start start 2023-11-08T08:21:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f9e0d4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9f9806b7b0 0x7f9f98072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f9e0d4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f980726c0 con 0x7f9f980b0880 2023-11-08T08:21:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f9e0d4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f980b43f0 con 0x7f9f980ad1f0 2023-11-08T08:21:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f9e0d4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9806bc50 con 0x7f9f9806b7b0 2023-11-08T08:21:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f977fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9f980b0880 0x7f9f980aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53326/0 (socket says 172.21.15.116:53326) 2023-11-08T08:21:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f977fe700 1 -- 172.21.15.116:0/1450558876 learned_addr learned my addr 172.21.15.116:0/1450558876 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697999802 0 0) 0x7f9f980726c0 con 0x7f9f980b0880 2023-11-08T08:21:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f800035b0 con 0x7f9f980b0880 2023-11-08T08:21:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f96ffd700 1 --2- 172.21.15.116:0/1450558876 >> v2:172.21.15.129:3300/0 conn(0x7f9f9806b7b0 0x7f9f98072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713791832 0 0) 0x7f9f800035b0 con 0x7f9f980b0880 2023-11-08T08:21:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 >> v2:172.21.15.129:3300/0 conn(0x7f9f9806b7b0 msgr2=0x7f9f98072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 --2- 172.21.15.116:0/1450558876 >> v2:172.21.15.129:3300/0 conn(0x7f9f9806b7b0 0x7f9f98072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 >> v1:172.21.15.129:6789/0 conn(0x7f9f980ad1f0 legacy=0x7f9f980ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980b46d0 con 0x7f9f980b0880 2023-11-08T08:21:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9f8c001670 con 0x7f9f980b0880 2023-11-08T08:21:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 >> v1:172.21.15.116:6789/0 conn(0x7f9f980b0880 legacy=0x7f9f980aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80003a00 0x7f9f80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f80004850 0x7f9f80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f80005200 0x7f9f8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9806bc50 con 0x7f9f80005200 2023-11-08T08:21:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000c080 con 0x7f9f80003a00 2023-11-08T08:21:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.711+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000c240 con 0x7f9f80004850 2023-11-08T08:21:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f80005200 0x7f9f8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f977fe700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80003a00 0x7f9f80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f96ffd700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f80004850 0x7f9f80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 -- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f80004850 msgr2=0x7f9f80004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f80004850 0x7f9f80004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 -- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80003a00 msgr2=0x7f9f80003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80003a00 0x7f9f80003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 -- 172.21.15.116:0/1450558876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800035b0 con 0x7f9f80005200 2023-11-08T08:21:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 -- 172.21.15.116:0/1450558876 reap_dead start 2023-11-08T08:21:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f97fff700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f80005200 0x7f9f8000ba10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000c190 tx=0x7f9f80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86ae5fe629b779b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90001db0 con 0x7f9f80005200 2023-11-08T08:21:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9000ccf0 con 0x7f9f80005200 2023-11-08T08:21:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.712+0000 7f9f967fc700 1 -- 172.21.15.116:0/1450558876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90003c40 con 0x7f9f80005200 2023-11-08T08:21:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/1450558876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f80005200 msgr2=0x7f9f8000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f80005200 0x7f9f8000ba10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000c190 tx=0x7f9f80005880 comp rx=0 tx=0).stop 2023-11-08T08:21:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/1450558876 shutdown_connections 2023-11-08T08:21:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/1450558876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f80005200 0x7f9f8000ba10 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/1450558876 >> 172.21.15.116:0/1450558876 conn(0x7f9f980b2fc0 msgr2=0x7f9f98071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/1450558876 shutdown_connections 2023-11-08T08:21:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.713+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/1450558876 wait complete. 2023-11-08T08:21:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.714+0000 7f9f9e0d4700 1 Processor -- start 2023-11-08T08:21:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.714+0000 7f9f9e0d4700 1 -- start start 2023-11-08T08:21:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80005200 0x7f9f98154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 0x7f9f980c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f980c7b10 0x7f9f980c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f98155450 con 0x7f9f9815f570 2023-11-08T08:21:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980c6e90 con 0x7f9f80005200 2023-11-08T08:21:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980c7170 con 0x7f9f980c7b10 2023-11-08T08:21:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 0x7f9f980c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 0x7f9f980c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53892/0 (socket says 172.21.15.116:53892) 2023-11-08T08:21:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 -- 172.21.15.116:0/2157548606 learned_addr learned my addr 172.21.15.116:0/2157548606 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f977fe700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80005200 0x7f9f98154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f97fff700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f980c7b10 0x7f9f980c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 -- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f980c7b10 msgr2=0x7f9f980c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f980c7b10 0x7f9f980c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 -- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80005200 msgr2=0x7f9f98154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80005200 0x7f9f98154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 -- 172.21.15.116:0/2157548606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980b4fe0 con 0x7f9f9815f570 2023-11-08T08:21:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f96ffd700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 0x7f9f980c6520 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9f880119a0 tx=0x7f9f88011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8800f730 con 0x7f9f9815f570 2023-11-08T08:21:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8800f890 con 0x7f9f9815f570 2023-11-08T08:21:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f980b4d00 con 0x7f9f9815f570 2023-11-08T08:21:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8800fc20 con 0x7f9f9815f570 2023-11-08T08:21:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.715+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f980b49b0 con 0x7f9f9815f570 2023-11-08T08:21:18.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.717+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9f8800fd80 con 0x7f9f9815f570 2023-11-08T08:21:18.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.717+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9815d0a0 con 0x7f9f9815f570 2023-11-08T08:21:18.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.717+0000 7f9f7ffff700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f840682f0 0x7f9f8406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.717+0000 7f9f977fe700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f840682f0 0x7f9f8406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.717+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9f880863e0 con 0x7f9f9815f570 2023-11-08T08:21:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.718+0000 7f9f977fe700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f840682f0 0x7f9f8406a7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c001c90 tx=0x7f9f8c001b20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.722+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f9815d0a0 con 0x7f9f9815f570 2023-11-08T08:21:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.946+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f9815d0a0 con 0x7f9f9815f570 2023-11-08T08:21:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.946+0000 7f9f7ffff700 1 -- 172.21.15.116:0/2157548606 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7f9f9815d0a0 con 0x7f9f9815f570 2023-11-08T08:21:18.948 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:18.948 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:21:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f840682f0 msgr2=0x7f9f8406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f840682f0 0x7f9f8406a7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c001c90 tx=0x7f9f8c001b20 comp rx=0 tx=0).stop 2023-11-08T08:21:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 msgr2=0x7f9f980c6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 0x7f9f980c6520 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9f880119a0 tx=0x7f9f88011cb0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 shutdown_connections 2023-11-08T08:21:18.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9f980c7b10 0x7f9f980c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9f840682f0 0x7f9f8406a7a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f80005200 0x7f9f98154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 --2- 172.21.15.116:0/2157548606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9815f570 0x7f9f980c6520 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:18.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.948+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 >> 172.21.15.116:0/2157548606 conn(0x7f9f980b2fc0 msgr2=0x7f9f9806c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:18.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.949+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 shutdown_connections 2023-11-08T08:21:18.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:18.949+0000 7f9f9e0d4700 1 -- 172.21.15.116:0/2157548606 wait complete. 2023-11-08T08:21:18.952 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:21:18.961 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.080+0000 7fa4cc8da700 1 Processor -- start 2023-11-08T08:21:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.080+0000 7fa4cc8da700 1 -- start start 2023-11-08T08:21:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4cc8da700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c806a270 0x7fa4c806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4cc8da700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa4c806e570 0x7fa4c8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4cc8da700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c806ab80 con 0x7fa4c815cc80 2023-11-08T08:21:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4cc8da700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c80b2490 con 0x7fa4c806e570 2023-11-08T08:21:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4cc8da700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c806e9b0 con 0x7fa4c806a270 2023-11-08T08:21:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4c6d9d700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa4c806e570 0x7fa4c8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4c659c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c806a270 0x7fa4c806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4c5d9b700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa4c815cc80 0x7fa4c806de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:37950/0 (socket says 172.21.15.116:37950) 2023-11-08T08:21:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.081+0000 7fa4c5d9b700 1 -- 172.21.15.116:0/2611468944 learned_addr learned my addr 172.21.15.116:0/2611468944 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c559a700 1 -- 172.21.15.116:0/2611468944 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927565178 0 0) 0x7fa4c806ab80 con 0x7fa4c815cc80 2023-11-08T08:21:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c559a700 1 -- 172.21.15.116:0/2611468944 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b00035b0 con 0x7fa4c815cc80 2023-11-08T08:21:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c659c700 1 -- 172.21.15.116:0/2611468944 >> v2:172.21.15.129:3300/0 conn(0x7fa4c806e570 msgr2=0x7fa4c8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c659c700 1 --2- 172.21.15.116:0/2611468944 >> v2:172.21.15.129:3300/0 conn(0x7fa4c806e570 0x7fa4c8070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c659c700 1 -- 172.21.15.116:0/2611468944 >> v1:172.21.15.129:6789/0 conn(0x7fa4c815cc80 legacy=0x7fa4c806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c659c700 1 -- 172.21.15.116:0/2611468944 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c80b1810 con 0x7fa4c806a270 2023-11-08T08:21:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c6d9d700 1 --2- 172.21.15.116:0/2611468944 >> v2:172.21.15.129:3300/0 conn(0x7fa4c806e570 0x7fa4c8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c659c700 1 --2- 172.21.15.116:0/2611468944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c806a270 0x7fa4c806a640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009840 tx=0x7fa4bc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b370730e6d2761 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c559a700 1 -- 172.21.15.116:0/2611468944 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc011830 con 0x7fa4c806a270 2023-11-08T08:21:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c559a700 1 -- 172.21.15.116:0/2611468944 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc011990 con 0x7fa4c806a270 2023-11-08T08:21:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4c559a700 1 -- 172.21.15.116:0/2611468944 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc011d00 con 0x7fa4c806a270 2023-11-08T08:21:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4cc8da700 1 -- 172.21.15.116:0/2611468944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c806a270 msgr2=0x7fa4c806a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.082+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/2611468944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c806a270 0x7fa4c806a640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009840 tx=0x7fa4bc009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:19.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.083+0000 7fa4cc8da700 1 -- 172.21.15.116:0/2611468944 shutdown_connections 2023-11-08T08:21:19.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.083+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/2611468944 >> v2:172.21.15.129:3300/0 conn(0x7fa4c806e570 0x7fa4c8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.083+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/2611468944 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c806a270 0x7fa4c806a640 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.083+0000 7fa4cc8da700 1 -- 172.21.15.116:0/2611468944 >> 172.21.15.116:0/2611468944 conn(0x7fa4c806c010 msgr2=0x7fa4c80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:19.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.083+0000 7fa4cc8da700 1 -- 172.21.15.116:0/2611468944 shutdown_connections 2023-11-08T08:21:19.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.083+0000 7fa4cc8da700 1 -- 172.21.15.116:0/2611468944 wait complete. 2023-11-08T08:21:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.084+0000 7fa4cc8da700 1 Processor -- start 2023-11-08T08:21:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.084+0000 7fa4cc8da700 1 -- start start 2023-11-08T08:21:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4cc8da700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c806a270 0x7fa4c80cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4cc8da700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 0x7fa4c8152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4cc8da700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c815cc80 0x7fa4c80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4cc8da700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80cbd10 con 0x7fa4c806e570 2023-11-08T08:21:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4cc8da700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80c7110 con 0x7fa4c806a270 2023-11-08T08:21:19.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4cc8da700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c80cad30 con 0x7fa4c815cc80 2023-11-08T08:21:19.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4c5d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 0x7fa4c8152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4c5d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 0x7fa4c8152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53902/0 (socket says 172.21.15.116:53902) 2023-11-08T08:21:19.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4c5d9b700 1 -- 172.21.15.116:0/643839771 learned_addr learned my addr 172.21.15.116:0/643839771 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:19.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4c6d9d700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c815cc80 0x7fa4c80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.085+0000 7fa4c659c700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c806a270 0x7fa4c80cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c5d9b700 1 -- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c815cc80 msgr2=0x7fa4c80b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c5d9b700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c815cc80 0x7fa4c80b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c5d9b700 1 -- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c806a270 msgr2=0x7fa4c80cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c5d9b700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c806a270 0x7fa4c80cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c5d9b700 1 -- 172.21.15.116:0/643839771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0009a70 con 0x7fa4c806e570 2023-11-08T08:21:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c6d9d700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c815cc80 0x7fa4c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c659c700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c806a270 0x7fa4c80cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4c5d9b700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 0x7fa4c8152b40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c4d0 tx=0x7fa4b800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8005d70 con 0x7fa4c806e570 2023-11-08T08:21:19.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b80070d0 con 0x7fa4c806e570 2023-11-08T08:21:19.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc0094a0 con 0x7fa4c806e570 2023-11-08T08:21:19.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8007440 con 0x7fa4c806e570 2023-11-08T08:21:19.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.086+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c80b1dd0 con 0x7fa4c806e570 2023-11-08T08:21:19.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.087+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c80b1af0 con 0x7fa4c806e570 2023-11-08T08:21:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.087+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa4b8019030 con 0x7fa4c806e570 2023-11-08T08:21:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.088+0000 7fa4b6ffd700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4a80682f0 0x7fa4a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.088+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa4b8084f10 con 0x7fa4c806e570 2023-11-08T08:21:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.088+0000 7fa4c659c700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4a80682f0 0x7fa4a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.088+0000 7fa4c659c700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4a80682f0 0x7fa4a806a7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc000c00 tx=0x7fa4bc01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.092+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4b804b5e0 con 0x7fa4c806e570 2023-11-08T08:21:19.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:21:19.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:21:19.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.322+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4c80b20b0 con 0x7fa4c806e570 2023-11-08T08:21:19.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.322+0000 7fa4b6ffd700 1 -- 172.21.15.116:0/643839771 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4654 (secure 0 0 0) 0x7fa4b8017020 con 0x7fa4c806e570 2023-11-08T08:21:19.324 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:19.324 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:20:33.012833+0000","modified":"2023-11-08T08:20:36.023322+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5781":{"gid":5781,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/409645101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":409645101},{"type":"v1","addr":"172.21.15.129:6834","nonce":409645101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-08T08:21:19.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4a80682f0 msgr2=0x7fa4a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4a80682f0 0x7fa4a806a7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc000c00 tx=0x7fa4bc01c040 comp rx=0 tx=0).stop 2023-11-08T08:21:19.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 msgr2=0x7fa4c8152b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 0x7fa4c8152b40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c4d0 tx=0x7fa4b800c890 comp rx=0 tx=0).stop 2023-11-08T08:21:19.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 shutdown_connections 2023-11-08T08:21:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa4c815cc80 0x7fa4c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa4a80682f0 0x7fa4a806a7a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c806a270 0x7fa4c80cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 --2- 172.21.15.116:0/643839771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4c806e570 0x7fa4c8152b40 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 >> 172.21.15.116:0/643839771 conn(0x7fa4c806c010 msgr2=0x7fa4c80ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 shutdown_connections 2023-11-08T08:21:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.324+0000 7fa4cc8da700 1 -- 172.21.15.116:0/643839771 wait complete. 2023-11-08T08:21:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-08T08:21:19.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.456+0000 7f47bf3ef700 1 Processor -- start 2023-11-08T08:21:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.456+0000 7f47bf3ef700 1 -- start start 2023-11-08T08:21:19.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bf3ef700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47b80ce040 0x7f47b815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bf3ef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bf3ef700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47b80b1770 0x7f47b80b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bf3ef700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b80b2130 con 0x7f47b80ce040 2023-11-08T08:21:19.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bf3ef700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b813fc90 con 0x7f47b80b1770 2023-11-08T08:21:19.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bf3ef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b813f6d0 con 0x7f47b8142db0 2023-11-08T08:21:19.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bd18b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47b80ce040 0x7f47b815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bd18b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47b80ce040 0x7f47b815d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53904/0 (socket says 172.21.15.116:53904) 2023-11-08T08:21:19.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.457+0000 7f47bd18b700 1 -- 172.21.15.116:0/711393186 learned_addr learned my addr 172.21.15.116:0/711393186 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:19.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd98c700 1 --2- 172.21.15.116:0/711393186 >> v2:172.21.15.129:3300/0 conn(0x7f47b80b1770 0x7f47b80b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bc98a700 1 --2- 172.21.15.116:0/711393186 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd18b700 1 -- 172.21.15.116:0/711393186 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 msgr2=0x7f47b813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:19.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd18b700 1 --2- 172.21.15.116:0/711393186 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b813ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd18b700 1 -- 172.21.15.116:0/711393186 >> v2:172.21.15.129:3300/0 conn(0x7f47b80b1770 msgr2=0x7f47b80b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:19.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd18b700 1 --2- 172.21.15.116:0/711393186 >> v2:172.21.15.129:3300/0 conn(0x7f47b80b1770 0x7f47b80b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd18b700 1 -- 172.21.15.116:0/711393186 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b813f9b0 con 0x7f47b80ce040 2023-11-08T08:21:19.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bd18b700 1 --2- 172.21.15.116:0/711393186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80ce040 0x7f47b815d900 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f47b80c37c0 tx=0x7f47a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a2cff2677ba804 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47affff700 1 -- 172.21.15.116:0/711393186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800b870 con 0x7f47b80ce040 2023-11-08T08:21:19.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47affff700 1 -- 172.21.15.116:0/711393186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a800b9d0 con 0x7f47b80ce040 2023-11-08T08:21:19.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47affff700 1 -- 172.21.15.116:0/711393186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800bd60 con 0x7f47b80ce040 2023-11-08T08:21:19.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bf3ef700 1 -- 172.21.15.116:0/711393186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80ce040 msgr2=0x7f47b815d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/711393186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80ce040 0x7f47b815d900 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f47b80c37c0 tx=0x7f47a8009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.458+0000 7f47bf3ef700 1 -- 172.21.15.116:0/711393186 shutdown_connections 2023-11-08T08:21:19.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.459+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/711393186 >> v2:172.21.15.129:3300/0 conn(0x7f47b80b1770 0x7f47b80b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.459+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/711393186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80ce040 0x7f47b815d900 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.459+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/711393186 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.459+0000 7f47bf3ef700 1 -- 172.21.15.116:0/711393186 >> 172.21.15.116:0/711393186 conn(0x7f47b806c010 msgr2=0x7f47b80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:19.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.459+0000 7f47bf3ef700 1 -- 172.21.15.116:0/711393186 shutdown_connections 2023-11-08T08:21:19.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.459+0000 7f47bf3ef700 1 -- 172.21.15.116:0/711393186 wait complete. 2023-11-08T08:21:19.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.460+0000 7f47bf3ef700 1 Processor -- start 2023-11-08T08:21:19.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.460+0000 7f47bf3ef700 1 -- start start 2023-11-08T08:21:19.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bf3ef700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 0x7f47b814ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bf3ef700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b80ce040 0x7f47b80c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bf3ef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b815c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bf3ef700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80b59c0 con 0x7f47b80b1770 2023-11-08T08:21:19.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bf3ef700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b814ecd0 con 0x7f47b80ce040 2023-11-08T08:21:19.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bf3ef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b815c780 con 0x7f47b8142db0 2023-11-08T08:21:19.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 0x7f47b814ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 0x7f47b814ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53908/0 (socket says 172.21.15.116:53908) 2023-11-08T08:21:19.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 -- 172.21.15.116:0/1486084490 learned_addr learned my addr 172.21.15.116:0/1486084490 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:19.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 -- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 msgr2=0x7f47b815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 -- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b80ce040 msgr2=0x7f47b80c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bc98a700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b80ce040 0x7f47b80c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b80ce040 0x7f47b80c2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bd18b700 1 -- 172.21.15.116:0/1486084490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009550 con 0x7f47b80b1770 2023-11-08T08:21:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.461+0000 7f47bc98a700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b80ce040 0x7f47b80c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.462+0000 7f47bd18b700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 0x7f47b814ee70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f47b8070850 tx=0x7f47a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.462+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8004310 con 0x7f47b80b1770 2023-11-08T08:21:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.462+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a801f070 con 0x7f47b80b1770 2023-11-08T08:21:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.462+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b815d3e0 con 0x7f47b80b1770 2023-11-08T08:21:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.462+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a80036a0 con 0x7f47b80b1770 2023-11-08T08:21:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.462+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b815d100 con 0x7f47b80b1770 2023-11-08T08:21:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.463+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b80ce410 con 0x7f47b80b1770 2023-11-08T08:21:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.464+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47a802a020 con 0x7f47b80b1770 2023-11-08T08:21:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.464+0000 7f47adffb700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f479c0682f0 0x7f479c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.465+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f47a80877c0 con 0x7f47b80b1770 2023-11-08T08:21:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.465+0000 7f47bc98a700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f479c0682f0 0x7f479c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.465+0000 7f47bc98a700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f479c0682f0 0x7f479c06a7a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47b81f3f20 tx=0x7f47b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.468+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47a8054040 con 0x7f47b80b1770 2023-11-08T08:21:19.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.653+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47b80743f0 con 0x7f47b80b1770 2023-11-08T08:21:19.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.654+0000 7f47adffb700 1 -- 172.21.15.116:0/1486084490 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+17909 (secure 0 0 0) 0x7f47a8057660 con 0x7f47b80b1770 2023-11-08T08:21:19.656 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:19.656 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":44,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:20:34.014710+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:20:30.822342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-11-08T08:20:31.988683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f479c0682f0 msgr2=0x7f479c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f479c0682f0 0x7f479c06a7a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47b81f3f20 tx=0x7f47b4008040 comp rx=0 tx=0).stop 2023-11-08T08:21:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 msgr2=0x7f47b814ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 0x7f47b814ee70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f47b8070850 tx=0x7f47a8004750 comp rx=0 tx=0).stop 2023-11-08T08:21:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 shutdown_connections 2023-11-08T08:21:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47b8142db0 0x7f47b815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b80b1770 0x7f47b814ee70 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f479c0682f0 0x7f479c06a7a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 --2- 172.21.15.116:0/1486084490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b80ce040 0x7f47b80c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.656+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 >> 172.21.15.116:0/1486084490 conn(0x7f47b806c010 msgr2=0x7f47b8158b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.657+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 shutdown_connections 2023-11-08T08:21:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.657+0000 7f47bf3ef700 1 -- 172.21.15.116:0/1486084490 wait complete. 2023-11-08T08:21:19.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-08T08:21:19.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.790+0000 7f8ca6d72700 1 Processor -- start 2023-11-08T08:21:19.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.790+0000 7f8ca6d72700 1 -- start start 2023-11-08T08:21:19.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8ca6d72700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ca006a980 0x7f8ca00cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8ca6d72700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca00cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8ca6d72700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca015e0d0 con 0x7f8ca015a4d0 2023-11-08T08:21:19.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8ca6d72700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca00ce560 con 0x7f8ca006a980 2023-11-08T08:21:19.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8ca6d72700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca015e320 con 0x7f8ca00cdd50 2023-11-08T08:21:19.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ca006a980 0x7f8ca00cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ca006a980 0x7f8ca00cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53914/0 (socket says 172.21.15.116:53914) 2023-11-08T08:21:19.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 -- 172.21.15.116:0/2693596108 learned_addr learned my addr 172.21.15.116:0/2693596108 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:19.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8ca530f700 1 --2- 172.21.15.116:0/2693596108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca00cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 -- 172.21.15.116:0/2693596108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 msgr2=0x7f8ca00cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:19.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 --2- 172.21.15.116:0/2693596108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca00cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 -- 172.21.15.116:0/2693596108 >> v1:172.21.15.129:6789/0 conn(0x7f8ca015a4d0 legacy=0x7f8ca006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:19.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 -- 172.21.15.116:0/2693596108 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca015e600 con 0x7f8ca006a980 2023-11-08T08:21:19.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9ffff700 1 --2- 172.21.15.116:0/2693596108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca00cd810 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0098f0 tx=0x7f8c8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6617aef1215f95a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9f7fe700 1 -- 172.21.15.116:0/2693596108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c004230 con 0x7f8ca006a980 2023-11-08T08:21:19.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9f7fe700 1 -- 172.21.15.116:0/2693596108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c004390 con 0x7f8ca006a980 2023-11-08T08:21:19.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.791+0000 7f8c9f7fe700 1 -- 172.21.15.116:0/2693596108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0036a0 con 0x7f8ca006a980 2023-11-08T08:21:19.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 -- 172.21.15.116:0/2693596108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 msgr2=0x7f8ca00cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/2693596108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca00cd810 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0098f0 tx=0x7f8c8c009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:19.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 -- 172.21.15.116:0/2693596108 shutdown_connections 2023-11-08T08:21:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/2693596108 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca00cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/2693596108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca00cd810 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 -- 172.21.15.116:0/2693596108 >> 172.21.15.116:0/2693596108 conn(0x7f8ca006c010 msgr2=0x7f8ca00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 -- 172.21.15.116:0/2693596108 shutdown_connections 2023-11-08T08:21:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.792+0000 7f8ca6d72700 1 -- 172.21.15.116:0/2693596108 wait complete. 2023-11-08T08:21:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.793+0000 7f8ca6d72700 1 Processor -- start 2023-11-08T08:21:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.793+0000 7f8ca6d72700 1 -- start start 2023-11-08T08:21:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca6d72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca01f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca6d72700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca6d72700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ca015a4d0 0x7f8ca015d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca6d72700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca01f2c30 con 0x7f8ca006a980 2023-11-08T08:21:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca6d72700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca006e1f0 con 0x7f8ca015a4d0 2023-11-08T08:21:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca6d72700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca006d670 con 0x7f8ca00cdd50 2023-11-08T08:21:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca01f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca01f26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53930/0 (socket says 172.21.15.116:53930) 2023-11-08T08:21:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 -- 172.21.15.116:0/3942046906 learned_addr learned my addr 172.21.15.116:0/3942046906 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:19.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8c9ffff700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 -- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 msgr2=0x7f8ca015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca530f700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ca015a4d0 0x7f8ca015d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca015b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 -- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ca015a4d0 msgr2=0x7f8ca015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:19.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ca015a4d0 0x7f8ca015d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:19.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8ca4b0e700 1 -- 172.21.15.116:0/3942046906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c009550 con 0x7f8ca006a980 2023-11-08T08:21:19.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.794+0000 7f8c9ffff700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:19.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.795+0000 7f8ca4b0e700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca01f26f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400b4a0 tx=0x7f8c9400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.795+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94011830 con 0x7f8ca006a980 2023-11-08T08:21:19.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.795+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c94011990 con 0x7f8ca006a980 2023-11-08T08:21:19.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.795+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94011d40 con 0x7f8ca006a980 2023-11-08T08:21:19.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.795+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca00ce1b0 con 0x7f8ca006a980 2023-11-08T08:21:19.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.795+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca015e8e0 con 0x7f8ca006a980 2023-11-08T08:21:19.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.796+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca015a8a0 con 0x7f8ca006a980 2023-11-08T08:21:19.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.797+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c9400f050 con 0x7f8ca006a980 2023-11-08T08:21:19.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.797+0000 7f8c9d7fa700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8c840682f0 0x7f8c8406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:19.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.797+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8c94085e30 con 0x7f8ca006a980 2023-11-08T08:21:19.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.797+0000 7f8c9ffff700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8c840682f0 0x7f8c8406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:19.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.798+0000 7f8c9ffff700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8c840682f0 0x7f8c8406a7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00cd350 tx=0x7f8c8c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:19.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:19.801+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c94052620 con 0x7f8ca006a980 2023-11-08T08:21:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.024+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8ca00ad0e0 con 0x7f8ca006a980 2023-11-08T08:21:20.530 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-08T08:21:20.529+0000 7f50f4469700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-08T08:21:20.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.536+0000 7f8c9d7fa700 1 -- 172.21.15.116:0/3942046906 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8c94055c40 con 0x7f8ca006a980 2023-11-08T08:21:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8c840682f0 msgr2=0x7f8c8406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8c840682f0 0x7f8c8406a7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00cd350 tx=0x7f8c8c008030 comp rx=0 tx=0).stop 2023-11-08T08:21:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 msgr2=0x7f8ca01f26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca01f26f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400b4a0 tx=0x7f8c9400b860 comp rx=0 tx=0).stop 2023-11-08T08:21:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 shutdown_connections 2023-11-08T08:21:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8c840682f0 0x7f8c8406a7a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ca015a4d0 0x7f8ca015d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca006a980 0x7f8ca01f26f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 --2- 172.21.15.116:0/3942046906 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8ca00cdd50 0x7f8ca015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 >> 172.21.15.116:0/3942046906 conn(0x7f8ca006c010 msgr2=0x7f8ca0154840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 shutdown_connections 2023-11-08T08:21:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.539+0000 7f8ca6d72700 1 -- 172.21.15.116:0/3942046906 wait complete. 2023-11-08T08:21:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-08T08:21:20.544 INFO:tasks.ceph.mds.b.smithi129.stderr: -4470> 2023-11-08T08:20:38.614+0000 7f9ff5884700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:21:20.544 INFO:tasks.ceph.mds.b.smithi129.stderr: -2968> 2023-11-08T08:20:42.051+0000 7f9ff207d700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5922=0-4194304@1} caps={5922=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5922 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55eecefe9700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-08T08:21:20.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-08T08:21:20.573 INFO:tasks.ceph.mds.b.smithi129.stdout:starting mds.b at 2023-11-08T08:21:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.683+0000 7f3223968700 1 Processor -- start 2023-11-08T08:21:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.683+0000 7f3223968700 1 -- start start 2023-11-08T08:21:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.684+0000 7f3223968700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f321c0b50d0 0x7f321c06c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.684+0000 7f3223968700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 0x7f321c06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.684+0000 7f3223968700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0cd480 con 0x7f321c15a590 2023-11-08T08:21:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.684+0000 7f3223968700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321c0cd760 con 0x7f321c0b50d0 2023-11-08T08:21:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.684+0000 7f3223968700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0cda40 con 0x7f321c15a960 2023-11-08T08:21:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 0x7f321c06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 0x7f321c06c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:41256/0 (socket says 172.21.15.116:41256) 2023-11-08T08:21:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 -- 172.21.15.116:0/158563057 learned_addr learned my addr 172.21.15.116:0/158563057 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3220f03700 1 --2- 172.21.15.116:0/158563057 >> v2:172.21.15.129:3300/0 conn(0x7f321c0b50d0 0x7f321c06c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 -- 172.21.15.116:0/158563057 >> v2:172.21.15.129:3300/0 conn(0x7f321c0b50d0 msgr2=0x7f321c06c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 --2- 172.21.15.116:0/158563057 >> v2:172.21.15.129:3300/0 conn(0x7f321c0b50d0 0x7f321c06c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 -- 172.21.15.116:0/158563057 >> v1:172.21.15.129:6789/0 conn(0x7f321c15a590 legacy=0x7f321c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 -- 172.21.15.116:0/158563057 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c0cdd20 con 0x7f321c15a960 2023-11-08T08:21:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3221f05700 1 --2- 172.21.15.116:0/158563057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 0x7f321c06c980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3214009e50 tx=0x7f321400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ef5693667d8ccc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3213fff700 1 -- 172.21.15.116:0/158563057 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400e070 con 0x7f321c15a960 2023-11-08T08:21:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3213fff700 1 -- 172.21.15.116:0/158563057 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321400b910 con 0x7f321c15a960 2023-11-08T08:21:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.685+0000 7f3213fff700 1 -- 172.21.15.116:0/158563057 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400bcc0 con 0x7f321c15a960 2023-11-08T08:21:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 -- 172.21.15.116:0/158563057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 msgr2=0x7f321c06c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 --2- 172.21.15.116:0/158563057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 0x7f321c06c980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3214009e50 tx=0x7f321400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 -- 172.21.15.116:0/158563057 shutdown_connections 2023-11-08T08:21:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 --2- 172.21.15.116:0/158563057 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c15a960 0x7f321c06c980 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 --2- 172.21.15.116:0/158563057 >> v2:172.21.15.129:3300/0 conn(0x7f321c0b50d0 0x7f321c06c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 -- 172.21.15.116:0/158563057 >> 172.21.15.116:0/158563057 conn(0x7f321c0cf400 msgr2=0x7f321c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 -- 172.21.15.116:0/158563057 shutdown_connections 2023-11-08T08:21:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.686+0000 7f3223968700 1 -- 172.21.15.116:0/158563057 wait complete. 2023-11-08T08:21:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.687+0000 7f3223968700 1 Processor -- start 2023-11-08T08:21:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.687+0000 7f3223968700 1 -- start start 2023-11-08T08:21:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c0b50d0 0x7f321c1f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f321c15a590 0x7f321c14b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 0x7f321c0c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0c90e0 con 0x7f321c15a960 2023-11-08T08:21:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c158ab0 con 0x7f321c15a590 2023-11-08T08:21:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0c6c50 con 0x7f321c0b50d0 2023-11-08T08:21:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 0x7f321c0c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 0x7f321c0c2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53944/0 (socket says 172.21.15.116:53944) 2023-11-08T08:21:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 -- 172.21.15.116:0/2612760040 learned_addr learned my addr 172.21.15.116:0/2612760040 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221704700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c0b50d0 0x7f321c1f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3220f03700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f321c15a590 0x7f321c14b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 -- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c0b50d0 msgr2=0x7f321c1f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c0b50d0 0x7f321c1f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 -- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f321c15a590 msgr2=0x7f321c14b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f321c15a590 0x7f321c14b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 -- 172.21.15.116:0/2612760040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009b30 con 0x7f321c15a960 2023-11-08T08:21:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3221f05700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 0x7f321c0c2f30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f321c0cdb80 tx=0x7f3214004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214004800 con 0x7f321c15a960 2023-11-08T08:21:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3214004960 con 0x7f321c15a960 2023-11-08T08:21:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.688+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c06a350 con 0x7f321c15a960 2023-11-08T08:21:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.689+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321401b670 con 0x7f321c15a960 2023-11-08T08:21:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.689+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c0ce000 con 0x7f321c15a960 2023-11-08T08:21:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.690+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c06a4e0 con 0x7f321c15a960 2023-11-08T08:21:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.690+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f321402b070 con 0x7f321c15a960 2023-11-08T08:21:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.690+0000 7f3211ffb700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32000682f0 0x7f320006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.691+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3214087310 con 0x7f321c15a960 2023-11-08T08:21:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.691+0000 7f3221704700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32000682f0 0x7f320006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.691+0000 7f3221704700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32000682f0 0x7f320006a7a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f320c007fd0 tx=0x7f320c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.695+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3214053b10 con 0x7f321c15a960 2023-11-08T08:21:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:20.919+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f321c068450 con 0x7f321c15a960 2023-11-08T08:21:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.538+0000 7f3211ffb700 1 -- 172.21.15.116:0/2612760040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3214057130 con 0x7f321c15a960 2023-11-08T08:21:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32000682f0 msgr2=0x7f320006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32000682f0 0x7f320006a7a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f320c007fd0 tx=0x7f320c009040 comp rx=0 tx=0).stop 2023-11-08T08:21:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 msgr2=0x7f321c0c2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 0x7f321c0c2f30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f321c0cdb80 tx=0x7f3214004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 shutdown_connections 2023-11-08T08:21:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f321c15a960 0x7f321c0c2f30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f321c0b50d0 0x7f321c1f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32000682f0 0x7f320006a7a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 --2- 172.21.15.116:0/2612760040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f321c15a590 0x7f321c14b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 >> 172.21.15.116:0/2612760040 conn(0x7f321c0cf400 msgr2=0x7f321c0b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:21.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 shutdown_connections 2023-11-08T08:21:21.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.541+0000 7f3223968700 1 -- 172.21.15.116:0/2612760040 wait complete. 2023-11-08T08:21:21.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-08T08:21:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 Processor -- start 2023-11-08T08:21:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 -- start start 2023-11-08T08:21:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f018c15c9d0 0x7f018c071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 0x7f018c15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c0d0b50 con 0x7f018c0cdae0 2023-11-08T08:21:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f018c15cda0 con 0x7f018c15c9d0 2023-11-08T08:21:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.678+0000 7f01925b5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c15de10 con 0x7f018c15cf50 2023-11-08T08:21:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 0x7f018c15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018bfff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f018c15c9d0 0x7f018c071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f0190b52700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f018c0cdae0 0x7f018c0cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:37972/0 (socket says 172.21.15.116:37972) 2023-11-08T08:21:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 0x7f018c15d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:41270/0 (socket says 172.21.15.116:41270) 2023-11-08T08:21:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f0190b52700 1 -- 172.21.15.116:0/2220939704 learned_addr learned my addr 172.21.15.116:0/2220939704 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 -- 172.21.15.116:0/2220939704 >> v2:172.21.15.129:3300/0 conn(0x7f018c15c9d0 msgr2=0x7f018c071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 --2- 172.21.15.116:0/2220939704 >> v2:172.21.15.129:3300/0 conn(0x7f018c15c9d0 0x7f018c071790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 -- 172.21.15.116:0/2220939704 >> v1:172.21.15.129:6789/0 conn(0x7f018c0cdae0 legacy=0x7f018c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 -- 172.21.15.116:0/2220939704 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c15d8e0 con 0x7f018c15cf50 2023-11-08T08:21:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018bfff700 1 --2- 172.21.15.116:0/2220939704 >> v2:172.21.15.129:3300/0 conn(0x7f018c15c9d0 0x7f018c071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018b7fe700 1 --2- 172.21.15.116:0/2220939704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 0x7f018c15d3a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0180009840 tx=0x7f0180009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e14e8ffc54875a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018affd700 1 -- 172.21.15.116:0/2220939704 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0180011830 con 0x7f018c15cf50 2023-11-08T08:21:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.679+0000 7f018affd700 1 -- 172.21.15.116:0/2220939704 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0180011990 con 0x7f018c15cf50 2023-11-08T08:21:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f018affd700 1 -- 172.21.15.116:0/2220939704 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0180011d20 con 0x7f018c15cf50 2023-11-08T08:21:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 -- 172.21.15.116:0/2220939704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 msgr2=0x7f018c15d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 --2- 172.21.15.116:0/2220939704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 0x7f018c15d3a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0180009840 tx=0x7f0180009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 -- 172.21.15.116:0/2220939704 shutdown_connections 2023-11-08T08:21:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 --2- 172.21.15.116:0/2220939704 >> v2:172.21.15.129:3300/0 conn(0x7f018c15c9d0 0x7f018c071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 --2- 172.21.15.116:0/2220939704 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c15cf50 0x7f018c15d3a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 -- 172.21.15.116:0/2220939704 >> 172.21.15.116:0/2220939704 conn(0x7f018c0b24a0 msgr2=0x7f018c0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 -- 172.21.15.116:0/2220939704 shutdown_connections 2023-11-08T08:21:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.680+0000 7f01925b5700 1 -- 172.21.15.116:0/2220939704 wait complete. 2023-11-08T08:21:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.681+0000 7f01925b5700 1 Processor -- start 2023-11-08T08:21:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.681+0000 7f01925b5700 1 -- start start 2023-11-08T08:21:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f01925b5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c0cdae0 0x7f018c14c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f01925b5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f018c15c9d0 0x7f018c15ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f01925b5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 0x7f018c15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f01925b5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0b1080 con 0x7f018c15cf50 2023-11-08T08:21:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f01925b5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0b0470 con 0x7f018c15c9d0 2023-11-08T08:21:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f01925b5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0b0750 con 0x7f018c0cdae0 2023-11-08T08:21:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f0190b52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 0x7f018c15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f0190b52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 0x7f018c15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53946/0 (socket says 172.21.15.116:53946) 2023-11-08T08:21:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f0190b52700 1 -- 172.21.15.116:0/2785163756 learned_addr learned my addr 172.21.15.116:0/2785163756 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f018bfff700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c0cdae0 0x7f018c14c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:21.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.682+0000 7f018b7fe700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f018c15c9d0 0x7f018c15ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:21.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0190b52700 1 -- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c0cdae0 msgr2=0x7f018c14c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:21.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0190b52700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c0cdae0 0x7f018c14c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0190b52700 1 -- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f018c15c9d0 msgr2=0x7f018c15ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:21.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0190b52700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f018c15c9d0 0x7f018c15ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:21.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0190b52700 1 -- 172.21.15.116:0/2785163756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c0094a0 con 0x7f018c15cf50 2023-11-08T08:21:21.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0190b52700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 0x7f018c15b3a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f018400cbc0 tx=0x7f018400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:21.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018400a9e0 con 0x7f018c15cf50 2023-11-08T08:21:21.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018400ab40 con 0x7f018c15cf50 2023-11-08T08:21:21.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01800094a0 con 0x7f018c15cf50 2023-11-08T08:21:21.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184012400 con 0x7f018c15cf50 2023-11-08T08:21:21.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.683+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c070cb0 con 0x7f018c15cf50 2023-11-08T08:21:21.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.684+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c15db30 con 0x7f018c15cf50 2023-11-08T08:21:21.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.684+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0184019030 con 0x7f018c15cf50 2023-11-08T08:21:21.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.685+0000 7f0188ff9700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f01780682f0 0x7f017806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:21.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.685+0000 7f018bfff700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f01780682f0 0x7f017806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:21.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.685+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f01840857e0 con 0x7f018c15cf50 2023-11-08T08:21:21.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.685+0000 7f018bfff700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f01780682f0 0x7f017806a7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f018c0af630 tx=0x7f017c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:21.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.689+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f018404b5e0 con 0x7f018c15cf50 2023-11-08T08:21:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:21.869+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f018c070f90 con 0x7f018c15cf50 2023-11-08T08:21:22.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.545+0000 7f0188ff9700 1 -- 172.21.15.116:0/2785163756 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0184051fa0 con 0x7f018c15cf50 2023-11-08T08:21:22.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f01780682f0 msgr2=0x7f017806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f01780682f0 0x7f017806a7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f018c0af630 tx=0x7f017c009210 comp rx=0 tx=0).stop 2023-11-08T08:21:22.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 msgr2=0x7f018c15b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 0x7f018c15b3a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f018400cbc0 tx=0x7f018400ced0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 shutdown_connections 2023-11-08T08:21:22.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f018c15cf50 0x7f018c15b3a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f018c0cdae0 0x7f018c14c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f01780682f0 0x7f017806a7a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 --2- 172.21.15.116:0/2785163756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f018c15c9d0 0x7f018c15ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 >> 172.21.15.116:0/2785163756 conn(0x7f018c0b24a0 msgr2=0x7f018c0c55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:22.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 shutdown_connections 2023-11-08T08:21:22.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.548+0000 7f01925b5700 1 -- 172.21.15.116:0/2785163756 wait complete. 2023-11-08T08:21:22.552 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2023-11-08T08:21:22.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-08T08:21:22.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.689+0000 7f906b3c9700 1 Processor -- start 2023-11-08T08:21:22.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.689+0000 7f906b3c9700 1 -- start start 2023-11-08T08:21:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f906b3c9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f90640ccf40 0x7f90640cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f906b3c9700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90640cb420 con 0x7f90640ae7a0 2023-11-08T08:21:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f906b3c9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90640cb700 con 0x7f906415c620 2023-11-08T08:21:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f906b3c9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9064066620 con 0x7f90640ccf40 2023-11-08T08:21:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f9068964700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f90640ae7a0 0x7f9064160920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47704/0 (socket says 172.21.15.116:47704) 2023-11-08T08:21:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f9068964700 1 -- 172.21.15.116:0/2471859879 learned_addr learned my addr 172.21.15.116:0/2471859879 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004328906 0 0) 0x7f90640cb420 con 0x7f90640ae7a0 2023-11-08T08:21:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90500035b0 con 0x7f90640ae7a0 2023-11-08T08:21:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f9069966700 1 --2- 172.21.15.116:0/2471859879 >> v2:172.21.15.129:3300/0 conn(0x7f90640ccf40 0x7f90640cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990417709 0 0) 0x7f90500035b0 con 0x7f90640ae7a0 2023-11-08T08:21:22.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 >> v2:172.21.15.129:3300/0 conn(0x7f90640ccf40 msgr2=0x7f90640cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:22.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 --2- 172.21.15.116:0/2471859879 >> v2:172.21.15.129:3300/0 conn(0x7f90640ccf40 0x7f90640cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 >> v1:172.21.15.129:6789/0 conn(0x7f906415c620 legacy=0x7f90640cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90640cb9e0 con 0x7f90640ae7a0 2023-11-08T08:21:22.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9060001670 con 0x7f90640ae7a0 2023-11-08T08:21:22.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 >> v1:172.21.15.116:6789/0 conn(0x7f90640ae7a0 legacy=0x7f9064160920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.690+0000 7f905bfff700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050003a00 0x7f9050003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9050004850 0x7f9050004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9050005200 0x7f905000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9064066620 con 0x7f9050005200 2023-11-08T08:21:22.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905000c080 con 0x7f9050004850 2023-11-08T08:21:22.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905000c240 con 0x7f9050003a00 2023-11-08T08:21:22.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9050005200 0x7f905000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9068964700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9050004850 0x7f9050004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069165700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050003a00 0x7f9050003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 -- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050003a00 msgr2=0x7f9050003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:22.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050003a00 0x7f9050003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 -- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9050004850 msgr2=0x7f9050004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:22.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9050004850 0x7f9050004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 -- 172.21.15.116:0/2471859879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90500035b0 con 0x7f9050005200 2023-11-08T08:21:22.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069165700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050003a00 0x7f9050003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:22.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 -- 172.21.15.116:0/2471859879 reap_dead start 2023-11-08T08:21:22.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f9069966700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9050005200 0x7f905000ba10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f905c009fc0 tx=0x7f9050005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473e96c71f40ba33 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:22.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c003a20 con 0x7f9050005200 2023-11-08T08:21:22.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c00f040 con 0x7f9050005200 2023-11-08T08:21:22.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.691+0000 7f905bfff700 1 -- 172.21.15.116:0/2471859879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c003de0 con 0x7f9050005200 2023-11-08T08:21:22.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 -- 172.21.15.116:0/2471859879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9050005200 msgr2=0x7f905000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9050005200 0x7f905000ba10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f905c009fc0 tx=0x7f9050005880 comp rx=0 tx=0).stop 2023-11-08T08:21:22.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 -- 172.21.15.116:0/2471859879 shutdown_connections 2023-11-08T08:21:22.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 --2- 172.21.15.116:0/2471859879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9050005200 0x7f905000ba10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 -- 172.21.15.116:0/2471859879 >> 172.21.15.116:0/2471859879 conn(0x7f90640cf370 msgr2=0x7f90640cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:22.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 -- 172.21.15.116:0/2471859879 shutdown_connections 2023-11-08T08:21:22.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.692+0000 7f906b3c9700 1 -- 172.21.15.116:0/2471859879 wait complete. 2023-11-08T08:21:22.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.693+0000 7f906b3c9700 1 Processor -- start 2023-11-08T08:21:22.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.693+0000 7f906b3c9700 1 -- start start 2023-11-08T08:21:22.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050005200 0x7f906414c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 0x7f90640711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90640cbdb0 0x7f90640cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90640cc770 con 0x7f906415d720 2023-11-08T08:21:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906415dc20 con 0x7f90640cbdb0 2023-11-08T08:21:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906415df00 con 0x7f9050005200 2023-11-08T08:21:22.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 0x7f90640711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 0x7f90640711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47770/0 (socket says 172.21.15.116:47770) 2023-11-08T08:21:22.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 -- 172.21.15.116:0/3681423360 learned_addr learned my addr 172.21.15.116:0/3681423360 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:22.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9069165700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050005200 0x7f906414c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9069966700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90640cbdb0 0x7f90640cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 -- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050005200 msgr2=0x7f906414c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050005200 0x7f906414c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 -- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90640cbdb0 msgr2=0x7f90640cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:22.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90640cbdb0 0x7f90640cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:22.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 -- 172.21.15.116:0/3681423360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906415c9f0 con 0x7f906415d720 2023-11-08T08:21:22.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9069165700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050005200 0x7f906414c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:22.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9068964700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 0x7f90640711f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f90600017a0 tx=0x7f906000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:22.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906000c5f0 con 0x7f906415d720 2023-11-08T08:21:22.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906000c750 con 0x7f906415d720 2023-11-08T08:21:22.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9064160400 con 0x7f906415d720 2023-11-08T08:21:22.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906000cb00 con 0x7f906415d720 2023-11-08T08:21:22.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.694+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906415cb80 con 0x7f906415d720 2023-11-08T08:21:22.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.695+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f906000cc60 con 0x7f906415d720 2023-11-08T08:21:22.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.695+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90640743f0 con 0x7f906415d720 2023-11-08T08:21:22.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.696+0000 7f9059ffb700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f90480682f0 0x7f904806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:22.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.696+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5710+0+0 (secure 0 0 0) 0x7f9060085ab0 con 0x7f906415d720 2023-11-08T08:21:22.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.696+0000 7f9069165700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f90480682f0 0x7f904806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:22.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.696+0000 7f9069165700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f90480682f0 0x7f904806a7a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9054000c00 tx=0x7f9054009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:22.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.700+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9060052470 con 0x7f906415d720 2023-11-08T08:21:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:22.882+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9064066190 con 0x7f906415d720 2023-11-08T08:21:23.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.554+0000 7f9059ffb700 1 -- 172.21.15.116:0/3681423360 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9060055a90 con 0x7f906415d720 2023-11-08T08:21:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f90480682f0 msgr2=0x7f904806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f90480682f0 0x7f904806a7a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9054000c00 tx=0x7f9054009040 comp rx=0 tx=0).stop 2023-11-08T08:21:23.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 msgr2=0x7f90640711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 0x7f90640711f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f90600017a0 tx=0x7f906000eb80 comp rx=0 tx=0).stop 2023-11-08T08:21:23.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 shutdown_connections 2023-11-08T08:21:23.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f90480682f0 0x7f904806a7a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90640cbdb0 0x7f90640cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9050005200 0x7f906414c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 --2- 172.21.15.116:0/3681423360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f906415d720 0x7f90640711f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 >> 172.21.15.116:0/3681423360 conn(0x7f90640cf370 msgr2=0x7f90640cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:23.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 shutdown_connections 2023-11-08T08:21:23.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.557+0000 7f906b3c9700 1 -- 172.21.15.116:0/3681423360 wait complete. 2023-11-08T08:21:23.564 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2023-11-08T08:21:23.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-08T08:21:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.698+0000 7fd7e248a700 1 Processor -- start 2023-11-08T08:21:23.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.698+0000 7fd7e248a700 1 -- start start 2023-11-08T08:21:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7e248a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7dc0b1650 0x7fd7dc0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7e248a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc1599b0 0x7fd7dc15d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7e248a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc159ec0 con 0x7fd7dc0af0a0 2023-11-08T08:21:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7e248a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7dc15aa60 con 0x7fd7dc0b1650 2023-11-08T08:21:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7e248a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0b1f60 con 0x7fd7dc1599b0 2023-11-08T08:21:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7dc0b1650 0x7fd7dc0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7dc0b1650 0x7fd7dc0b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47784/0 (socket says 172.21.15.116:47784) 2023-11-08T08:21:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 -- 172.21.15.116:0/4217417105 learned_addr learned my addr 172.21.15.116:0/4217417105 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 -- 172.21.15.116:0/4217417105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc1599b0 msgr2=0x7fd7dc15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 --2- 172.21.15.116:0/4217417105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc1599b0 0x7fd7dc15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 -- 172.21.15.116:0/4217417105 >> v1:172.21.15.129:6789/0 conn(0x7fd7dc0af0a0 legacy=0x7fd7dc15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 -- 172.21.15.116:0/4217417105 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc15d8a0 con 0x7fd7dc0b1650 2023-11-08T08:21:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.699+0000 7fd7dbfff700 1 --2- 172.21.15.116:0/4217417105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc0b1650 0x7fd7dc0b1a20 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0098f0 tx=0x7fd7cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e206b33ac3aa787 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7daffd700 1 -- 172.21.15.116:0/4217417105 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc004230 con 0x7fd7dc0b1650 2023-11-08T08:21:23.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7daffd700 1 -- 172.21.15.116:0/4217417105 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc004390 con 0x7fd7dc0b1650 2023-11-08T08:21:23.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7daffd700 1 -- 172.21.15.116:0/4217417105 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0036a0 con 0x7fd7dc0b1650 2023-11-08T08:21:23.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 -- 172.21.15.116:0/4217417105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc0b1650 msgr2=0x7fd7dc0b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 --2- 172.21.15.116:0/4217417105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc0b1650 0x7fd7dc0b1a20 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0098f0 tx=0x7fd7cc009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:23.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 -- 172.21.15.116:0/4217417105 shutdown_connections 2023-11-08T08:21:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 --2- 172.21.15.116:0/4217417105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc0b1650 0x7fd7dc0b1a20 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 --2- 172.21.15.116:0/4217417105 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc1599b0 0x7fd7dc15d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 -- 172.21.15.116:0/4217417105 >> 172.21.15.116:0/4217417105 conn(0x7fd7dc06c010 msgr2=0x7fd7dc0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 -- 172.21.15.116:0/4217417105 shutdown_connections 2023-11-08T08:21:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.700+0000 7fd7e248a700 1 -- 172.21.15.116:0/4217417105 wait complete. 2023-11-08T08:21:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.701+0000 7fd7e248a700 1 Processor -- start 2023-11-08T08:21:23.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.701+0000 7fd7e248a700 1 -- start start 2023-11-08T08:21:23.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e248a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7dc0af0a0 0x7fd7dc0b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:23.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e248a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc0b1650 0x7fd7dc0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:23.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e248a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 0x7fd7dc151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:23.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e248a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0b59f0 con 0x7fd7dc147b10 2023-11-08T08:21:23.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e248a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc14f490 con 0x7fd7dc0af0a0 2023-11-08T08:21:23.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e248a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0b33c0 con 0x7fd7dc0b1650 2023-11-08T08:21:23.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e0a27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 0x7fd7dc151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e0a27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 0x7fd7dc151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47792/0 (socket says 172.21.15.116:47792) 2023-11-08T08:21:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.702+0000 7fd7e0a27700 1 -- 172.21.15.116:0/2930636849 learned_addr learned my addr 172.21.15.116:0/2930636849 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7dbfff700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7dc0af0a0 0x7fd7dc0b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7db7fe700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc0b1650 0x7fd7dc0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e0a27700 1 -- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc0b1650 msgr2=0x7fd7dc0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e0a27700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc0b1650 0x7fd7dc0c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e0a27700 1 -- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7dc0af0a0 msgr2=0x7fd7dc0b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e0a27700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7dc0af0a0 0x7fd7dc0b5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e0a27700 1 -- 172.21.15.116:0/2930636849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc009550 con 0x7fd7dc147b10 2023-11-08T08:21:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e0a27700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 0x7fd7dc151330 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400b990 tx=0x7fd7d400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d400c9d0 con 0x7fd7dc147b10 2023-11-08T08:21:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d400cb30 con 0x7fd7dc147b10 2023-11-08T08:21:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc15de60 con 0x7fd7dc147b10 2023-11-08T08:21:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4014440 con 0x7fd7dc147b10 2023-11-08T08:21:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.703+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc15db80 con 0x7fd7dc147b10 2023-11-08T08:21:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.704+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc15d590 con 0x7fd7dc147b10 2023-11-08T08:21:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.704+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd7d401b030 con 0x7fd7dc147b10 2023-11-08T08:21:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.705+0000 7fd7d8ff9700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7c80682f0 0x7fd7c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.705+0000 7fd7dbfff700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7c80682f0 0x7fd7c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.705+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fd7d4085630 con 0x7fd7dc147b10 2023-11-08T08:21:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.705+0000 7fd7dbfff700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7c80682f0 0x7fd7c806a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0b5070 tx=0x7fd7cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.709+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7d4052190 con 0x7fd7dc147b10 2023-11-08T08:21:23.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.889+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd7dc14f5d0 con 0x7fd7dc147b10 2023-11-08T08:21:23.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.893+0000 7fd7d8ff9700 1 -- 172.21.15.116:0/2930636849 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd7dc14f5d0 con 0x7fd7dc147b10 2023-11-08T08:21:23.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7c80682f0 msgr2=0x7fd7c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7c80682f0 0x7fd7c806a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0b5070 tx=0x7fd7cc008030 comp rx=0 tx=0).stop 2023-11-08T08:21:23.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 msgr2=0x7fd7dc151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:23.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 0x7fd7dc151330 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400b990 tx=0x7fd7d400bd50 comp rx=0 tx=0).stop 2023-11-08T08:21:23.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 shutdown_connections 2023-11-08T08:21:23.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7dc147b10 0x7fd7dc151330 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd7c80682f0 0x7fd7c806a7a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7dc0af0a0 0x7fd7dc0b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 --2- 172.21.15.116:0/2930636849 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd7dc0b1650 0x7fd7dc0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:23.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 >> 172.21.15.116:0/2930636849 conn(0x7fd7dc06c010 msgr2=0x7fd7dc0b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:23.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 shutdown_connections 2023-11-08T08:21:23.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:23.895+0000 7fd7e248a700 1 -- 172.21.15.116:0/2930636849 wait complete. 2023-11-08T08:21:23.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2023-11-08T08:21:24.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.026+0000 7f05b02c5700 1 Processor -- start 2023-11-08T08:21:24.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.026+0000 7f05b02c5700 1 -- start start 2023-11-08T08:21:24.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05b02c5700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f05a8163430 0x7f05a8165810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05b02c5700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a8151a50 con 0x7f05a8070d90 2023-11-08T08:21:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05b02c5700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a815b440 con 0x7f05a806a270 2023-11-08T08:21:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05b02c5700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a815f0c0 con 0x7f05a8163430 2023-11-08T08:21:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05ae862700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f05a8163430 0x7f05a8165810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05ae862700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f05a8163430 0x7f05a8165810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47798/0 (socket says 172.21.15.116:47798) 2023-11-08T08:21:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05ae862700 1 -- 172.21.15.116:0/1161204643 learned_addr learned my addr 172.21.15.116:0/1161204643 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.027+0000 7f05ad860700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f05a8070d90 0x7f05a80711e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47706/0 (socket says 172.21.15.116:47706) 2023-11-08T08:21:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ae862700 1 -- 172.21.15.116:0/1161204643 >> v1:172.21.15.129:6789/0 conn(0x7f05a806a270 legacy=0x7f05a806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ae862700 1 -- 172.21.15.116:0/1161204643 >> v1:172.21.15.116:6789/0 conn(0x7f05a8070d90 legacy=0x7f05a80711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ae862700 1 -- 172.21.15.116:0/1161204643 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8071ac0 con 0x7f05a8163430 2023-11-08T08:21:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ae862700 1 --2- 172.21.15.116:0/1161204643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8163430 0x7f05a8165810 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009e50 tx=0x7f05a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c458a1f21deeaf server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ad05f700 1 -- 172.21.15.116:0/1161204643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a000e070 con 0x7f05a8163430 2023-11-08T08:21:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ad05f700 1 -- 172.21.15.116:0/1161204643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0003a20 con 0x7f05a8163430 2023-11-08T08:21:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05ad05f700 1 -- 172.21.15.116:0/1161204643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0003dd0 con 0x7f05a8163430 2023-11-08T08:21:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.028+0000 7f05b02c5700 1 -- 172.21.15.116:0/1161204643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8163430 msgr2=0x7f05a8165810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.029+0000 7f05b02c5700 1 --2- 172.21.15.116:0/1161204643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8163430 0x7f05a8165810 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009e50 tx=0x7f05a000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.029+0000 7f05b02c5700 1 -- 172.21.15.116:0/1161204643 shutdown_connections 2023-11-08T08:21:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.029+0000 7f05b02c5700 1 --2- 172.21.15.116:0/1161204643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8163430 0x7f05a8165810 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.029+0000 7f05b02c5700 1 -- 172.21.15.116:0/1161204643 >> 172.21.15.116:0/1161204643 conn(0x7f05a806c010 msgr2=0x7f05a80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.029+0000 7f05b02c5700 1 -- 172.21.15.116:0/1161204643 shutdown_connections 2023-11-08T08:21:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.029+0000 7f05b02c5700 1 -- 172.21.15.116:0/1161204643 wait complete. 2023-11-08T08:21:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.030+0000 7f05b02c5700 1 Processor -- start 2023-11-08T08:21:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.030+0000 7f05b02c5700 1 -- start start 2023-11-08T08:21:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.030+0000 7f05b02c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 0x7f05a8153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.030+0000 7f05b02c5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8070d90 0x7f05a80c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.030+0000 7f05b02c5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f05a8163430 0x7f05a80c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05b02c5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8153a80 con 0x7f05a806a270 2023-11-08T08:21:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05b02c5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81541a0 con 0x7f05a8070d90 2023-11-08T08:21:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05b02c5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80c4e70 con 0x7f05a8163430 2023-11-08T08:21:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 0x7f05a8153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 0x7f05a8153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47806/0 (socket says 172.21.15.116:47806) 2023-11-08T08:21:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 -- 172.21.15.116:0/3501164848 learned_addr learned my addr 172.21.15.116:0/3501164848 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ad860700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8070d90 0x7f05a80c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae862700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f05a8163430 0x7f05a80c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 -- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f05a8163430 msgr2=0x7f05a80c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f05a8163430 0x7f05a80c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 -- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8070d90 msgr2=0x7f05a80c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8070d90 0x7f05a80c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 -- 172.21.15.116:0/3501164848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0009b30 con 0x7f05a806a270 2023-11-08T08:21:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05ae061700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 0x7f05a8153c60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c500 tx=0x7f05a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4011790 con 0x7f05a806a270 2023-11-08T08:21:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a40118f0 con 0x7f05a806a270 2023-11-08T08:21:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a815b0d0 con 0x7f05a806a270 2023-11-08T08:21:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4011ca0 con 0x7f05a806a270 2023-11-08T08:21:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.031+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a815adf0 con 0x7f05a806a270 2023-11-08T08:21:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.032+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a815aac0 con 0x7f05a806a270 2023-11-08T08:21:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.033+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f05a4010050 con 0x7f05a806a270 2023-11-08T08:21:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.033+0000 7f059effd700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f058c0682f0 0x7f058c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.033+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f05a4085670 con 0x7f05a806a270 2023-11-08T08:21:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.033+0000 7f05ad860700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f058c0682f0 0x7f058c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.033+0000 7f05ad860700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f058c0682f0 0x7f058c06a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f05a8152d10 tx=0x7f0598007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.037+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05a40521d0 con 0x7f05a806a270 2023-11-08T08:21:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.293+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7f05a80c42d0 con 0x7f05a806a270 2023-11-08T08:21:24.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.557+0000 7f059effd700 1 -- 172.21.15.116:0/3501164848 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v153) v1 ==== 149+0+0 (secure 0 0 0) 0x7f05a40557f0 con 0x7f05a806a270 2023-11-08T08:21:24.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.559+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f058c0682f0 msgr2=0x7f058c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.559+0000 7f05b02c5700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f058c0682f0 0x7f058c06a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f05a8152d10 tx=0x7f0598007d50 comp rx=0 tx=0).stop 2023-11-08T08:21:24.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 msgr2=0x7f05a8153c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 0x7f05a8153c60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c500 tx=0x7f05a400c8c0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 shutdown_connections 2023-11-08T08:21:24.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f05a8163430 0x7f05a80c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a806a270 0x7f05a8153c60 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f058c0682f0 0x7f058c06a7a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 --2- 172.21.15.116:0/3501164848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8070d90 0x7f05a80c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 >> 172.21.15.116:0/3501164848 conn(0x7f05a806c010 msgr2=0x7f05a815b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:24.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 shutdown_connections 2023-11-08T08:21:24.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.560+0000 7f05b02c5700 1 -- 172.21.15.116:0/3501164848 wait complete. 2023-11-08T08:21:24.574 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2023-11-08T08:21:24.574 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2023-11-08T08:21:24.575 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:21:24.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2023-11-08T08:21:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.696+0000 7f1537b31700 1 Processor -- start 2023-11-08T08:21:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.696+0000 7f1537b31700 1 -- start start 2023-11-08T08:21:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f1537b31700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f15300ce040 0x7f153015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f1537b31700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f153013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f1537b31700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15300b1770 0x7f15300b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f1537b31700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300b2130 con 0x7f15300b1770 2023-11-08T08:21:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f1537b31700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153013fc90 con 0x7f15300ce040 2023-11-08T08:21:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f1537b31700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153013f6d0 con 0x7f1530142db0 2023-11-08T08:21:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15300b1770 0x7f15300b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15300b1770 0x7f15300b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47812/0 (socket says 172.21.15.116:47812) 2023-11-08T08:21:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 -- 172.21.15.116:0/587244662 learned_addr learned my addr 172.21.15.116:0/587244662 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15358cd700 1 --2- 172.21.15.116:0/587244662 >> v2:172.21.15.129:3300/0 conn(0x7f15300ce040 0x7f153015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 -- 172.21.15.116:0/587244662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 msgr2=0x7f153013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15350cc700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f153013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f153013ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 -- 172.21.15.116:0/587244662 >> v2:172.21.15.129:3300/0 conn(0x7f15300ce040 msgr2=0x7f153015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 --2- 172.21.15.116:0/587244662 >> v2:172.21.15.129:3300/0 conn(0x7f15300ce040 0x7f153015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 -- 172.21.15.116:0/587244662 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153013f9b0 con 0x7f15300b1770 2023-11-08T08:21:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15358cd700 1 --2- 172.21.15.116:0/587244662 >> v2:172.21.15.129:3300/0 conn(0x7f15300ce040 0x7f153015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15350cc700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f153013ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15360ce700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300b1770 0x7f15300b1bf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f15300cfc50 tx=0x7f152800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a5ae61d82350b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15348cb700 1 -- 172.21.15.116:0/587244662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152800e070 con 0x7f15300b1770 2023-11-08T08:21:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15348cb700 1 -- 172.21.15.116:0/587244662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152800b9c0 con 0x7f15300b1770 2023-11-08T08:21:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.697+0000 7f15348cb700 1 -- 172.21.15.116:0/587244662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152800bd70 con 0x7f15300b1770 2023-11-08T08:21:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 -- 172.21.15.116:0/587244662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300b1770 msgr2=0x7f15300b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300b1770 0x7f15300b1bf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f15300cfc50 tx=0x7f152800b330 comp rx=0 tx=0).stop 2023-11-08T08:21:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 -- 172.21.15.116:0/587244662 shutdown_connections 2023-11-08T08:21:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300b1770 0x7f15300b1bf0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 --2- 172.21.15.116:0/587244662 >> v2:172.21.15.129:3300/0 conn(0x7f15300ce040 0x7f153015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 --2- 172.21.15.116:0/587244662 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f153013ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 -- 172.21.15.116:0/587244662 >> 172.21.15.116:0/587244662 conn(0x7f153006c010 msgr2=0x7f15300cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 -- 172.21.15.116:0/587244662 shutdown_connections 2023-11-08T08:21:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.698+0000 7f1537b31700 1 -- 172.21.15.116:0/587244662 wait complete. 2023-11-08T08:21:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.699+0000 7f1537b31700 1 Processor -- start 2023-11-08T08:21:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.699+0000 7f1537b31700 1 -- start start 2023-11-08T08:21:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f1537b31700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15300b1770 0x7f15301e2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f1537b31700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 0x7f15300c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f1537b31700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f1530152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f1537b31700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301e4360 con 0x7f15300ce040 2023-11-08T08:21:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f1537b31700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301e3480 con 0x7f15300b1770 2023-11-08T08:21:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f1537b31700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15301e36e0 con 0x7f1530142db0 2023-11-08T08:21:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 0x7f15300c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 0x7f15300c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47826/0 (socket says 172.21.15.116:47826) 2023-11-08T08:21:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 -- 172.21.15.116:0/1082742909 learned_addr learned my addr 172.21.15.116:0/1082742909 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15360ce700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f1530152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15358cd700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15300b1770 0x7f15301e2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 -- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 msgr2=0x7f1530152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f1530152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 -- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15300b1770 msgr2=0x7f15301e2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15300b1770 0x7f15301e2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 -- 172.21.15.116:0/1082742909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1528009b30 con 0x7f15300ce040 2023-11-08T08:21:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15358cd700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15300b1770 0x7f15301e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.700+0000 7f15350cc700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 0x7f15300c3a90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f152c00c750 tx=0x7f152c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.701+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c011800 con 0x7f15300ce040 2023-11-08T08:21:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.701+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c011960 con 0x7f15300ce040 2023-11-08T08:21:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.701+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153015d3e0 con 0x7f15300ce040 2023-11-08T08:21:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.701+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c011cd0 con 0x7f15300ce040 2023-11-08T08:21:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.701+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153015d100 con 0x7f15300ce040 2023-11-08T08:21:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.702+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15300ce410 con 0x7f15300ce040 2023-11-08T08:21:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.702+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f152c020070 con 0x7f15300ce040 2023-11-08T08:21:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.702+0000 7f15267fc700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f15140682f0 0x7f151406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.703+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f152c085930 con 0x7f15300ce040 2023-11-08T08:21:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.703+0000 7f15358cd700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f15140682f0 0x7f151406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.703+0000 7f15358cd700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f15140682f0 0x7f151406a7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1520008940 tx=0x7f1520008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.707+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f152c0523c0 con 0x7f15300ce040 2023-11-08T08:21:24.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:24.963+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7f1530066a90 con 0x7f15300ce040 2023-11-08T08:21:25.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.557+0000 7f15267fc700 1 -- 172.21.15.116:0/1082742909 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v154) v1 ==== 172+0+0 (secure 0 0 0) 0x7f152c0559e0 con 0x7f15300ce040 2023-11-08T08:21:25.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f15140682f0 msgr2=0x7f151406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f15140682f0 0x7f151406a7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1520008940 tx=0x7f1520008040 comp rx=0 tx=0).stop 2023-11-08T08:21:25.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 msgr2=0x7f15300c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 0x7f15300c3a90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f152c00c750 tx=0x7f152c00cb10 comp rx=0 tx=0).stop 2023-11-08T08:21:25.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 shutdown_connections 2023-11-08T08:21:25.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1530142db0 0x7f1530152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f15140682f0 0x7f151406a7a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15300b1770 0x7f15301e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 --2- 172.21.15.116:0/1082742909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15300ce040 0x7f15300c3a90 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 >> 172.21.15.116:0/1082742909 conn(0x7f153006c010 msgr2=0x7f15300cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 shutdown_connections 2023-11-08T08:21:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.559+0000 7f1537b31700 1 -- 172.21.15.116:0/1082742909 wait complete. 2023-11-08T08:21:25.573 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-08T08:21:25.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.699+0000 7fc677936700 1 Processor -- start 2023-11-08T08:21:25.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.699+0000 7fc677936700 1 -- start start 2023-11-08T08:21:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc677936700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc67015c810 0x7fc670160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc677936700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc67006a270 0x7fc67006a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc677936700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67006abb0 con 0x7fc6700718e0 2023-11-08T08:21:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc677936700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67015cbe0 con 0x7fc67006a270 2023-11-08T08:21:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc677936700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6701607a0 con 0x7fc67015c810 2023-11-08T08:21:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc675ed3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc67006a270 0x7fc67006a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc674ed1700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc6700718e0 0x7fc67006c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47716/0 (socket says 172.21.15.116:47716) 2023-11-08T08:21:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc674ed1700 1 -- 172.21.15.116:0/3070907879 learned_addr learned my addr 172.21.15.116:0/3070907879 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc675ed3700 1 -- 172.21.15.116:0/3070907879 >> v2:172.21.15.129:3300/0 conn(0x7fc67015c810 msgr2=0x7fc670160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc6756d2700 1 --2- 172.21.15.116:0/3070907879 >> v2:172.21.15.129:3300/0 conn(0x7fc67015c810 0x7fc670160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc675ed3700 1 --2- 172.21.15.116:0/3070907879 >> v2:172.21.15.129:3300/0 conn(0x7fc67015c810 0x7fc670160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc675ed3700 1 -- 172.21.15.116:0/3070907879 >> v1:172.21.15.116:6789/0 conn(0x7fc6700718e0 legacy=0x7fc67006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc667fff700 1 -- 172.21.15.116:0/3070907879 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662333891 0 0) 0x7fc67006abb0 con 0x7fc6700718e0 2023-11-08T08:21:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc675ed3700 1 -- 172.21.15.116:0/3070907879 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc670160a50 con 0x7fc67006a270 2023-11-08T08:21:25.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc675ed3700 1 --2- 172.21.15.116:0/3070907879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67006a270 0x7fc67006a670 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc6700b27c0 tx=0x7fc66800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c232934c0574a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:25.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc667fff700 1 -- 172.21.15.116:0/3070907879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66800e070 con 0x7fc67006a270 2023-11-08T08:21:25.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc667fff700 1 -- 172.21.15.116:0/3070907879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66800bbd0 con 0x7fc67006a270 2023-11-08T08:21:25.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.700+0000 7fc667fff700 1 -- 172.21.15.116:0/3070907879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668012430 con 0x7fc67006a270 2023-11-08T08:21:25.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 -- 172.21.15.116:0/3070907879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67006a270 msgr2=0x7fc67006a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 --2- 172.21.15.116:0/3070907879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67006a270 0x7fc67006a670 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc6700b27c0 tx=0x7fc66800b330 comp rx=0 tx=0).stop 2023-11-08T08:21:25.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 -- 172.21.15.116:0/3070907879 shutdown_connections 2023-11-08T08:21:25.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 --2- 172.21.15.116:0/3070907879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67006a270 0x7fc67006a670 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 --2- 172.21.15.116:0/3070907879 >> v2:172.21.15.129:3300/0 conn(0x7fc67015c810 0x7fc670160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 -- 172.21.15.116:0/3070907879 >> 172.21.15.116:0/3070907879 conn(0x7fc67006c010 msgr2=0x7fc6700cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:25.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 -- 172.21.15.116:0/3070907879 shutdown_connections 2023-11-08T08:21:25.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.701+0000 7fc677936700 1 -- 172.21.15.116:0/3070907879 wait complete. 2023-11-08T08:21:25.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.702+0000 7fc677936700 1 Processor -- start 2023-11-08T08:21:25.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.702+0000 7fc677936700 1 -- start start 2023-11-08T08:21:25.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc677936700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc67006a270 0x7fc67015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc677936700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 0x7fc6701f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc677936700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc67015c810 0x7fc6701ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc677936700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f23b0 con 0x7fc6700718e0 2023-11-08T08:21:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc677936700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f2030 con 0x7fc67006a270 2023-11-08T08:21:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc677936700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f5b60 con 0x7fc67015c810 2023-11-08T08:21:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc674ed1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 0x7fc6701f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc674ed1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 0x7fc6701f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47850/0 (socket says 172.21.15.116:47850) 2023-11-08T08:21:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc674ed1700 1 -- 172.21.15.116:0/544120725 learned_addr learned my addr 172.21.15.116:0/544120725 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc6756d2700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc67006a270 0x7fc67015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.703+0000 7fc675ed3700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc67015c810 0x7fc6701ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc674ed1700 1 -- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc67015c810 msgr2=0x7fc6701ef2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc674ed1700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc67015c810 0x7fc6701ef2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc674ed1700 1 -- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc67006a270 msgr2=0x7fc67015bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc674ed1700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc67006a270 0x7fc67015bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc674ed1700 1 -- 172.21.15.116:0/544120725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668009a70 con 0x7fc6700718e0 2023-11-08T08:21:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc675ed3700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc67015c810 0x7fc6701ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc674ed1700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 0x7fc6701f5620 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0011e0 tx=0x7fc66c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c004c80 con 0x7fc6700718e0 2023-11-08T08:21:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c004de0 con 0x7fc6700718e0 2023-11-08T08:21:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc670161010 con 0x7fc6700718e0 2023-11-08T08:21:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c007310 con 0x7fc6700718e0 2023-11-08T08:21:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.704+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc670160d30 con 0x7fc6700718e0 2023-11-08T08:21:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.705+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6700ceb20 con 0x7fc6700718e0 2023-11-08T08:21:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.705+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc66c01a020 con 0x7fc6700718e0 2023-11-08T08:21:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.706+0000 7fc665ffb700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6540682f0 0x7fc65406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.706+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fc66c085590 con 0x7fc6700718e0 2023-11-08T08:21:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.706+0000 7fc6756d2700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6540682f0 0x7fc65406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.706+0000 7fc6756d2700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6540682f0 0x7fc65406a7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc660006fd0 tx=0x7fc660008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.710+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc66c0520f0 con 0x7fc6700718e0 2023-11-08T08:21:25.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.889+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc6701f24f0 con 0x7fc6700718e0 2023-11-08T08:21:25.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.893+0000 7fc665ffb700 1 -- 172.21.15.116:0/544120725 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc6701f24f0 con 0x7fc6700718e0 2023-11-08T08:21:25.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6540682f0 msgr2=0x7fc65406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6540682f0 0x7fc65406a7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc660006fd0 tx=0x7fc660008040 comp rx=0 tx=0).stop 2023-11-08T08:21:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 msgr2=0x7fc6701f5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 0x7fc6701f5620 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0011e0 tx=0x7fc66c00ca30 comp rx=0 tx=0).stop 2023-11-08T08:21:25.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 shutdown_connections 2023-11-08T08:21:25.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc67015c810 0x7fc6701ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6540682f0 0x7fc65406a7a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc67006a270 0x7fc67015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 --2- 172.21.15.116:0/544120725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6700718e0 0x7fc6701f5620 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:25.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 >> 172.21.15.116:0/544120725 conn(0x7fc67006c010 msgr2=0x7fc67006db30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:25.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 shutdown_connections 2023-11-08T08:21:25.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:25.895+0000 7fc677936700 1 -- 172.21.15.116:0/544120725 wait complete. 2023-11-08T08:21:25.908 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-08T08:21:25.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:21:25.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:21:25.997 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-08T08:21:25.998 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:25.998 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:26.017 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:21:26.017 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:26.018 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:26.039 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:26.039 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:26.040 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:26.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:21:26.077 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:26.077 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:26.102 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:26.103 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-08T08:21:26.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.233+0000 7f81d17a4700 1 Processor -- start 2023-11-08T08:21:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.233+0000 7f81d17a4700 1 -- start start 2023-11-08T08:21:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.233+0000 7f81d17a4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 0x7f81cc0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81d17a4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f81cc160e70 0x7f81cc15c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81d17a4700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc06e900 con 0x7f81cc06e400 2023-11-08T08:21:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81d17a4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81cc160390 con 0x7f81cc160e70 2023-11-08T08:21:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81d17a4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc160670 con 0x7f81cc0d0c60 2023-11-08T08:21:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81ca7fc700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f81cc06e400 0x7f81cc0d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47726/0 (socket says 172.21.15.116:47726) 2023-11-08T08:21:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81ca7fc700 1 -- 172.21.15.116:0/3363431210 learned_addr learned my addr 172.21.15.116:0/3363431210 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703675616 0 0) 0x7f81cc06e900 con 0x7f81cc06e400 2023-11-08T08:21:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b40035b0 con 0x7f81cc06e400 2023-11-08T08:21:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81caffd700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 0x7f81cc0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81cb7fe700 1 --2- 172.21.15.116:0/3363431210 >> v2:172.21.15.129:3300/0 conn(0x7f81cc160e70 0x7f81cc15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 687625952 0 0) 0x7f81b40035b0 con 0x7f81cc06e400 2023-11-08T08:21:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 msgr2=0x7f81cc0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 0x7f81cc0b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 >> v2:172.21.15.129:3300/0 conn(0x7f81cc160e70 msgr2=0x7f81cc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 --2- 172.21.15.116:0/3363431210 >> v2:172.21.15.129:3300/0 conn(0x7f81cc160e70 0x7f81cc15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc160950 con 0x7f81cc06e400 2023-11-08T08:21:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f81c0001670 con 0x7f81cc06e400 2023-11-08T08:21:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 >> v1:172.21.15.116:6789/0 conn(0x7f81cc06e400 legacy=0x7f81cc0d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4003a00 0x7f81b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 0x7f81b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b4004d90 0x7f81b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc160390 con 0x7f81b4003a00 2023-11-08T08:21:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc160670 con 0x7f81b4004d90 2023-11-08T08:21:26.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b400bac0 con 0x7f81cc0d0c60 2023-11-08T08:21:26.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81caffd700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4003a00 0x7f81b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81cb7fe700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b4004d90 0x7f81b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.234+0000 7f81ca7fc700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 0x7f81b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81caffd700 1 -- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 msgr2=0x7f81b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81caffd700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81cc0d0c60 0x7f81b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81caffd700 1 -- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b4004d90 msgr2=0x7f81b4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81caffd700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81b4004d90 0x7f81b4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81caffd700 1 -- 172.21.15.116:0/3363431210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40035b0 con 0x7f81b4003a00 2023-11-08T08:21:26.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81caffd700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4003a00 0x7f81b4003e10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009890 tx=0x7f81b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c4d70266f03961 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:26.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00c070 con 0x7f81b4003a00 2023-11-08T08:21:26.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc013610 con 0x7f81b4003a00 2023-11-08T08:21:26.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81c9ffb700 1 -- 172.21.15.116:0/3363431210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0139a0 con 0x7f81b4003a00 2023-11-08T08:21:26.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81d17a4700 1 -- 172.21.15.116:0/3363431210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4003a00 msgr2=0x7f81b4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81d17a4700 1 --2- 172.21.15.116:0/3363431210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4003a00 0x7f81b4003e10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009890 tx=0x7f81b4005880 comp rx=0 tx=0).stop 2023-11-08T08:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81cb7fe700 1 -- 172.21.15.116:0/3363431210 reap_dead start 2023-11-08T08:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81d17a4700 1 -- 172.21.15.116:0/3363431210 shutdown_connections 2023-11-08T08:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81d17a4700 1 -- 172.21.15.116:0/3363431210 >> 172.21.15.116:0/3363431210 conn(0x7f81cc06a270 msgr2=0x7f81cc06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.235+0000 7f81d17a4700 1 -- 172.21.15.116:0/3363431210 shutdown_connections 2023-11-08T08:21:26.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.236+0000 7f81d17a4700 1 -- 172.21.15.116:0/3363431210 wait complete. 2023-11-08T08:21:26.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.237+0000 7f81d17a4700 1 Processor -- start 2023-11-08T08:21:26.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.237+0000 7f81d17a4700 1 -- start start 2023-11-08T08:21:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81d17a4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4003a00 0x7f81cc1f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81d17a4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 0x7f81cc143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81d17a4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81cc15fcf0 0x7f81cc14f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81d17a4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc14de80 con 0x7f81cc15d330 2023-11-08T08:21:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81d17a4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc15f860 con 0x7f81cc15fcf0 2023-11-08T08:21:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81d17a4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc1f6aa0 con 0x7f81b4003a00 2023-11-08T08:21:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81ca7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 0x7f81cc143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81ca7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 0x7f81cc143890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47866/0 (socket says 172.21.15.116:47866) 2023-11-08T08:21:26.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81ca7fc700 1 -- 172.21.15.116:0/2293593908 learned_addr learned my addr 172.21.15.116:0/2293593908 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:26.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81cb7fe700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81cc15fcf0 0x7f81cc14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.238+0000 7f81caffd700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4003a00 0x7f81cc1f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81ca7fc700 1 -- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4003a00 msgr2=0x7f81cc1f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81ca7fc700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4003a00 0x7f81cc1f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81ca7fc700 1 -- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81cc15fcf0 msgr2=0x7f81cc14f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81ca7fc700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81cc15fcf0 0x7f81cc14f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81ca7fc700 1 -- 172.21.15.116:0/2293593908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0b5be0 con 0x7f81cc15d330 2023-11-08T08:21:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81ca7fc700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 0x7f81cc143890 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f81c0007370 tx=0x7f81c0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c000f850 con 0x7f81cc15d330 2023-11-08T08:21:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c000f9d0 con 0x7f81cc15d330 2023-11-08T08:21:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc0d0740 con 0x7f81cc15d330 2023-11-08T08:21:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c000fd80 con 0x7f81cc15d330 2023-11-08T08:21:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.239+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc0b5d70 con 0x7f81cc15d330 2023-11-08T08:21:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.240+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc15f9f0 con 0x7f81cc15d330 2023-11-08T08:21:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.240+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81c001d460 con 0x7f81cc15d330 2023-11-08T08:21:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.240+0000 7f81b37fe700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b80682f0 0x7f81b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.241+0000 7f81caffd700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b80682f0 0x7f81b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.241+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f81c0085c10 con 0x7f81cc15d330 2023-11-08T08:21:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.241+0000 7f81caffd700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b80682f0 0x7f81b806a7a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f81bc000c00 tx=0x7f81bc00c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.245+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81c0052640 con 0x7f81cc15d330 2023-11-08T08:21:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.468+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81cc0743f0 con 0x7f81cc15d330 2023-11-08T08:21:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.468+0000 7f81b37fe700 1 -- 172.21.15.116:0/2293593908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f81c0055c60 con 0x7f81cc15d330 2023-11-08T08:21:26.470 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:26.470 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1453131209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1453131209},{"type":"v1","addr":"172.21.15.116:6835","nonce":1453131209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1312956671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1312956671},{"type":"v1","addr":"172.21.15.116:6837","nonce":1312956671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/317033476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":317033476},{"type":"v1","addr":"172.21.15.129:6835","nonce":317033476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/612652132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":612652132},{"type":"v1","addr":"172.21.15.129:6834","nonce":612652132}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[]} 2023-11-08T08:21:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b80682f0 msgr2=0x7f81b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b80682f0 0x7f81b806a7a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f81bc000c00 tx=0x7f81bc00c000 comp rx=0 tx=0).stop 2023-11-08T08:21:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 msgr2=0x7f81cc143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 0x7f81cc143890 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f81c0007370 tx=0x7f81c0007680 comp rx=0 tx=0).stop 2023-11-08T08:21:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 shutdown_connections 2023-11-08T08:21:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f81b80682f0 0x7f81b806a7a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81cc15fcf0 0x7f81cc14f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f81b4003a00 0x7f81cc1f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 --2- 172.21.15.116:0/2293593908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81cc15d330 0x7f81cc143890 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 >> 172.21.15.116:0/2293593908 conn(0x7f81cc06a270 msgr2=0x7f81cc0b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 shutdown_connections 2023-11-08T08:21:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.470+0000 7f81d17a4700 1 -- 172.21.15.116:0/2293593908 wait complete. 2023-11-08T08:21:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 26 2023-11-08T08:21:26.484 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-08T08:21:26.484 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-08T08:21:26.484 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-08T08:21:26.485 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:21:26.485 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-08T08:21:26.485 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-08T08:21:26.486 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-08T08:21:26.486 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:21:26.486 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-08T08:21:26.487 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-08T08:21:26.487 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-08T08:21:26.487 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:21:26.487 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-08T08:21:26.488 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-08T08:21:26.488 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-08T08:21:26.488 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:21:26.494 INFO:tasks.ceph.mds.a:Stopped 2023-11-08T08:21:26.494 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-08T08:21:26.495 INFO:tasks.ceph.mds.c:Stopped 2023-11-08T08:21:26.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-08T08:21:26.499 INFO:tasks.ceph.mds.b:Stopped 2023-11-08T08:21:26.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-08T08:21:26.501 INFO:tasks.ceph.mds.d:Stopped 2023-11-08T08:21:26.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-08T08:21:26.538 INFO:tasks.ceph.mds.a:Started 2023-11-08T08:21:26.539 DEBUG:teuthology.parallel:result is None 2023-11-08T08:21:26.539 INFO:tasks.ceph.mds.c:Started 2023-11-08T08:21:26.540 DEBUG:teuthology.parallel:result is None 2023-11-08T08:21:26.542 INFO:tasks.ceph.mds.b:Started 2023-11-08T08:21:26.543 DEBUG:teuthology.parallel:result is None 2023-11-08T08:21:26.544 INFO:tasks.ceph.mds.d:Started 2023-11-08T08:21:26.544 DEBUG:teuthology.parallel:result is None 2023-11-08T08:21:26.544 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-08T08:21:26.618 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-08T08:21:26.618 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-08T08:21:26.632 INFO:tasks.ceph.mds.b.smithi129.stdout:starting mds.b at 2023-11-08T08:21:26.632 INFO:tasks.ceph.mds.d.smithi129.stdout:starting mds.d at 2023-11-08T08:21:26.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.674+0000 7ff510524700 1 Processor -- start 2023-11-08T08:21:26.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.674+0000 7ff510524700 1 -- start start 2023-11-08T08:21:26.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff510524700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff50815c8c0 0x7ff508160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff510524700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b5dc0 0x7ff50806ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff510524700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080d2d30 con 0x7ff50806b050 2023-11-08T08:21:26.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff510524700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50806b4d0 con 0x7ff50815c8c0 2023-11-08T08:21:26.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff510524700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080d0b40 con 0x7ff5080b5dc0 2023-11-08T08:21:26.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff50815c8c0 0x7ff508160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff50815c8c0 0x7ff508160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47886/0 (socket says 172.21.15.116:47886) 2023-11-08T08:21:26.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 -- 172.21.15.116:0/766443796 learned_addr learned my addr 172.21.15.116:0/766443796 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:26.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50dabf700 1 --2- 172.21.15.116:0/766443796 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b5dc0 0x7ff50806ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 -- 172.21.15.116:0/766443796 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b5dc0 msgr2=0x7ff50806ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 --2- 172.21.15.116:0/766443796 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b5dc0 0x7ff50806ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 -- 172.21.15.116:0/766443796 >> v1:172.21.15.129:6789/0 conn(0x7ff50806b050 legacy=0x7ff5080d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 -- 172.21.15.116:0/766443796 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5081611f0 con 0x7ff50815c8c0 2023-11-08T08:21:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50e2c0700 1 --2- 172.21.15.116:0/766443796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50815c8c0 0x7ff508160cb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009e50 tx=0x7ff4f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a846f767f52075dd server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50d2be700 1 -- 172.21.15.116:0/766443796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f800e070 con 0x7ff50815c8c0 2023-11-08T08:21:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50d2be700 1 -- 172.21.15.116:0/766443796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f800b910 con 0x7ff50815c8c0 2023-11-08T08:21:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.675+0000 7ff50d2be700 1 -- 172.21.15.116:0/766443796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f800bcc0 con 0x7ff50815c8c0 2023-11-08T08:21:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 -- 172.21.15.116:0/766443796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50815c8c0 msgr2=0x7ff508160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 --2- 172.21.15.116:0/766443796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50815c8c0 0x7ff508160cb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009e50 tx=0x7ff4f800b330 comp rx=0 tx=0).stop 2023-11-08T08:21:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 -- 172.21.15.116:0/766443796 shutdown_connections 2023-11-08T08:21:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 --2- 172.21.15.116:0/766443796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50815c8c0 0x7ff508160cb0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 --2- 172.21.15.116:0/766443796 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5080b5dc0 0x7ff50806ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 -- 172.21.15.116:0/766443796 >> 172.21.15.116:0/766443796 conn(0x7ff50806c520 msgr2=0x7ff50806c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 -- 172.21.15.116:0/766443796 shutdown_connections 2023-11-08T08:21:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.676+0000 7ff510524700 1 -- 172.21.15.116:0/766443796 wait complete. 2023-11-08T08:21:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.677+0000 7ff510524700 1 Processor -- start 2023-11-08T08:21:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.677+0000 7ff510524700 1 -- start start 2023-11-08T08:21:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff510524700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff50806b050 0x7ff508153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff510524700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 0x7ff508157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff510524700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff50815c8c0 0x7ff508157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff510524700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080cc060 con 0x7ff5080b5dc0 2023-11-08T08:21:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff510524700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50815eca0 con 0x7ff50806b050 2023-11-08T08:21:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff510524700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080cbd80 con 0x7ff50815c8c0 2023-11-08T08:21:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 0x7ff508157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 0x7ff508157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47902/0 (socket says 172.21.15.116:47902) 2023-11-08T08:21:26.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 -- 172.21.15.116:0/3956716745 learned_addr learned my addr 172.21.15.116:0/3956716745 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:26.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50eac1700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff50815c8c0 0x7ff508157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50e2c0700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff50806b050 0x7ff508153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 -- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff50815c8c0 msgr2=0x7ff508157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff50815c8c0 0x7ff508157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 -- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff50806b050 msgr2=0x7ff508153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:26.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff50806b050 0x7ff508153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:26.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 -- 172.21.15.116:0/3956716745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8009b30 con 0x7ff5080b5dc0 2023-11-08T08:21:26.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff50dabf700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 0x7ff508157020 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00e9b0 tx=0x7ff4fc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:26.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.678+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc009710 con 0x7ff5080b5dc0 2023-11-08T08:21:26.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.679+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc00ee50 con 0x7ff5080b5dc0 2023-11-08T08:21:26.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.679+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5080cd760 con 0x7ff5080b5dc0 2023-11-08T08:21:26.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.679+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc00ca50 con 0x7ff5080b5dc0 2023-11-08T08:21:26.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.679+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080d2a50 con 0x7ff5080b5dc0 2023-11-08T08:21:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.679+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5080cda40 con 0x7ff5080b5dc0 2023-11-08T08:21:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.680+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4fc019030 con 0x7ff5080b5dc0 2023-11-08T08:21:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.680+0000 7ff506ffd700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4f40682f0 0x7ff4f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.680+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5296+0+0 (secure 0 0 0) 0x7ff4fc085810 con 0x7ff5080b5dc0 2023-11-08T08:21:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.680+0000 7ff50e2c0700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4f40682f0 0x7ff4f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:26.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.681+0000 7ff50e2c0700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4f40682f0 0x7ff4f406a7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800b370 tx=0x7ff4f8009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:26.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.684+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4fc052370 con 0x7ff5080b5dc0 2023-11-08T08:21:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:26.878+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff508067d70 con 0x7ff5080b5dc0 2023-11-08T08:21:27.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.672+0000 7ff506ffd700 1 -- 172.21.15.116:0/3956716745 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff4fc055990 con 0x7ff5080b5dc0 2023-11-08T08:21:27.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4f40682f0 msgr2=0x7ff4f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:27.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4f40682f0 0x7ff4f406a7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800b370 tx=0x7ff4f8009000 comp rx=0 tx=0).stop 2023-11-08T08:21:27.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 msgr2=0x7ff508157020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:27.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 0x7ff508157020 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00e9b0 tx=0x7ff4fc00ecc0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 shutdown_connections 2023-11-08T08:21:27.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff50815c8c0 0x7ff508157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff4f40682f0 0x7ff4f406a7a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff50806b050 0x7ff508153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 --2- 172.21.15.116:0/3956716745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5080b5dc0 0x7ff508157020 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 >> 172.21.15.116:0/3956716745 conn(0x7ff50806c520 msgr2=0x7ff5080b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 shutdown_connections 2023-11-08T08:21:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.675+0000 7ff510524700 1 -- 172.21.15.116:0/3956716745 wait complete. 2023-11-08T08:21:27.680 INFO:teuthology.orchestra.run.smithi116.stderr: removed all blocklist entries 2023-11-08T08:21:27.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-08T08:21:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.816+0000 7f386e0ef700 1 Processor -- start 2023-11-08T08:21:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 -- start start 2023-11-08T08:21:27.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f386806a270 0x7f386806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3868070d90 0x7f38680711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f386806dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3868151a50 con 0x7f386806a270 2023-11-08T08:21:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386815b440 con 0x7f3868070d90 2023-11-08T08:21:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f386e0ef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386806ab80 con 0x7f386806da40 2023-11-08T08:21:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.817+0000 7f38677fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f386806a270 0x7f386806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f386806a270 0x7f386806a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47916/0 (socket says 172.21.15.116:47916) 2023-11-08T08:21:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 -- 172.21.15.116:0/1618902011 learned_addr learned my addr 172.21.15.116:0/1618902011 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f3866ffd700 1 --2- 172.21.15.116:0/1618902011 >> v2:172.21.15.129:3300/0 conn(0x7f3868070d90 0x7f38680711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f3867fff700 1 --2- 172.21.15.116:0/1618902011 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f386806dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 -- 172.21.15.116:0/1618902011 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 msgr2=0x7f386806dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 --2- 172.21.15.116:0/1618902011 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f386806dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 -- 172.21.15.116:0/1618902011 >> v2:172.21.15.129:3300/0 conn(0x7f3868070d90 msgr2=0x7f38680711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 --2- 172.21.15.116:0/1618902011 >> v2:172.21.15.129:3300/0 conn(0x7f3868070d90 0x7f38680711e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 -- 172.21.15.116:0/1618902011 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868071ac0 con 0x7f386806a270 2023-11-08T08:21:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f3866ffd700 1 --2- 172.21.15.116:0/1618902011 >> v2:172.21.15.129:3300/0 conn(0x7f3868070d90 0x7f38680711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38677fe700 1 --2- 172.21.15.116:0/1618902011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f386806a270 0x7f386806a640 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f38680cd4f0 tx=0x7f3858009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6b55a059da4cdc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38667fc700 1 -- 172.21.15.116:0/1618902011 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385800b870 con 0x7f386806a270 2023-11-08T08:21:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38667fc700 1 -- 172.21.15.116:0/1618902011 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385800b9d0 con 0x7f386806a270 2023-11-08T08:21:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.818+0000 7f38667fc700 1 -- 172.21.15.116:0/1618902011 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385800bd60 con 0x7f386806a270 2023-11-08T08:21:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 -- 172.21.15.116:0/1618902011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f386806a270 msgr2=0x7f386806a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 --2- 172.21.15.116:0/1618902011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f386806a270 0x7f386806a640 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f38680cd4f0 tx=0x7f3858009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 -- 172.21.15.116:0/1618902011 shutdown_connections 2023-11-08T08:21:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 --2- 172.21.15.116:0/1618902011 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f386806dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 --2- 172.21.15.116:0/1618902011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f386806a270 0x7f386806a640 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 --2- 172.21.15.116:0/1618902011 >> v2:172.21.15.129:3300/0 conn(0x7f3868070d90 0x7f38680711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 -- 172.21.15.116:0/1618902011 >> 172.21.15.116:0/1618902011 conn(0x7f386806c010 msgr2=0x7f38680cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 -- 172.21.15.116:0/1618902011 shutdown_connections 2023-11-08T08:21:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.819+0000 7f386e0ef700 1 -- 172.21.15.116:0/1618902011 wait complete. 2023-11-08T08:21:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.820+0000 7f386e0ef700 1 Processor -- start 2023-11-08T08:21:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.820+0000 7f386e0ef700 1 -- start start 2023-11-08T08:21:27.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f386806a270 0x7f3868148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f38681485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 0x7f386814f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38681501c0 con 0x7f3868070d90 2023-11-08T08:21:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38680c49a0 con 0x7f386806a270 2023-11-08T08:21:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386814ff10 con 0x7f386806da40 2023-11-08T08:21:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 0x7f386814f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 0x7f386814f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47928/0 (socket says 172.21.15.116:47928) 2023-11-08T08:21:27.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 -- 172.21.15.116:0/193725134 learned_addr learned my addr 172.21.15.116:0/193725134 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:27.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f38677fe700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f386806a270 0x7f3868148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3866ffd700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f38681485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 -- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 msgr2=0x7f38681485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:27.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f38681485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 -- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f386806a270 msgr2=0x7f3868148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:27.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f386806a270 0x7f3868148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:27.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 -- 172.21.15.116:0/193725134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3858009550 con 0x7f3868070d90 2023-11-08T08:21:27.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f38677fe700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f386806a270 0x7f3868148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:27.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f3867fff700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 0x7f386814f9d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3860007e60 tx=0x7f386000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:27.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386000ede0 con 0x7f3868070d90 2023-11-08T08:21:27.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860014420 con 0x7f3868070d90 2023-11-08T08:21:27.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386815b0d0 con 0x7f3868070d90 2023-11-08T08:21:27.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600147d0 con 0x7f3868070d90 2023-11-08T08:21:27.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.821+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386815adf0 con 0x7f3868070d90 2023-11-08T08:21:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.823+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386815aac0 con 0x7f3868070d90 2023-11-08T08:21:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.823+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3860012050 con 0x7f3868070d90 2023-11-08T08:21:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.823+0000 7f385ffff700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f38480682f0 0x7f384806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.823+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3860086030 con 0x7f3868070d90 2023-11-08T08:21:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.823+0000 7f38677fe700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f38480682f0 0x7f384806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.824+0000 7f38677fe700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f38480682f0 0x7f384806a7a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3858000c00 tx=0x7f3858008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:27.827+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3860052a90 con 0x7f3868070d90 2023-11-08T08:21:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.080+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f38680743f0 con 0x7f3868070d90 2023-11-08T08:21:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.081+0000 7f385ffff700 1 -- 172.21.15.116:0/193725134 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f38600560b0 con 0x7f3868070d90 2023-11-08T08:21:28.083 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:28.083 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:28.083 INFO:teuthology.orchestra.run.smithi116.stdout: "auth_dump": [ 2023-11-08T08:21:28.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.083 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.a", 2023-11-08T08:21:28.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlwPmvARAAdJKHnQqKVPWnN7uQiHc+WQ==", 2023-11-08T08:21:28.084 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.084 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.084 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:21:28.085 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:21:28.085 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.086 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.b", 2023-11-08T08:21:28.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlXUbqDRAATJnFxEiS//echgEFyiS+iw==", 2023-11-08T08:21:28.086 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.086 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.087 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:21:28.087 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:21:28.087 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.088 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.c", 2023-11-08T08:21:28.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlNA7+CBAAkUCXKhY1qacysaUprxlU9w==", 2023-11-08T08:21:28.088 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.088 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.089 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:21:28.089 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:21:28.089 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.090 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.d", 2023-11-08T08:21:28.090 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtltlgoFRAAwAkFfezamgYHcC9awLpLpA==", 2023-11-08T08:21:28.090 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.091 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.091 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:21:28.091 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:21:28.091 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.092 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.0", 2023-11-08T08:21:28.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAyREtlwCT6MBAA0hUY6XxZE8Q/cW66704LAA==", 2023-11-08T08:21:28.092 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.092 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.093 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.093 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.094 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.1", 2023-11-08T08:21:28.094 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA1REtlHxOBLxAAHR12A9jsXL7W8TeG55VeWg==", 2023-11-08T08:21:28.094 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.095 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.095 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.095 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.096 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.2", 2023-11-08T08:21:28.096 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA4REtlcPidLhAAhNEoyldnbVRtP9IQr08wag==", 2023-11-08T08:21:28.096 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.097 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.097 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.097 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.098 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.3", 2023-11-08T08:21:28.098 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQA7REtlH/3XLBAASJ532Or0yrTC8qfr3kCziQ==", 2023-11-08T08:21:28.098 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.098 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.099 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.099 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.4", 2023-11-08T08:21:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBBREtl5+beLxAAXM67eNV2/FzHU4+gp3zMng==", 2023-11-08T08:21:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.5", 2023-11-08T08:21:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBEREtl4pChLhAA470kPHr4WFBGgwEi0CB95A==", 2023-11-08T08:21:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.6", 2023-11-08T08:21:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBHREtlh5g/LRAAY7eRgyAdWK0ExY6Qaf6hgQ==", 2023-11-08T08:21:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.7", 2023-11-08T08:21:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBKREtly78MKxAA5+d+0MekiMWQK1sAn1Ue6w==", 2023-11-08T08:21:28.106 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.106 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-08T08:21:28.106 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-08T08:21:28.106 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.107 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.0", 2023-11-08T08:21:28.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlHhh+GRAAKHMDRBv0YayhqERZiWXGAw==", 2023-11-08T08:21:28.108 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.108 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.108 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow r", 2023-11-08T08:21:28.108 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rw pool=cephfs_data" 2023-11-08T08:21:28.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.109 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.1", 2023-11-08T08:21:28.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtl2ICCIhAAdgVi5BOXs1xqGLKgTLEXoQ==", 2023-11-08T08:21:28.110 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.110 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.110 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:21:28.110 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:21:28.110 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:21:28.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.111 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.2", 2023-11-08T08:21:28.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtlKh8kHxAA/Ear8UA/wixquw0KBzet2A==", 2023-11-08T08:21:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:21:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:21:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:21:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.113 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.3", 2023-11-08T08:21:28.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAvREtl17vQJRAA/R0BvGRHgw8ZySwL4KyuGg==", 2023-11-08T08:21:28.113 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.114 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-08T08:21:28.114 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-08T08:21:28.114 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-08T08:21:28.114 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-08T08:21:28.115 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.115 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.admin", 2023-11-08T08:21:28.115 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtlwc+XHhAA8ruR6iJNLTqLSXROZ9iRHQ==", 2023-11-08T08:21:28.115 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.116 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:21:28.116 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-08T08:21:28.116 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-08T08:21:28.116 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.117 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.x", 2023-11-08T08:21:28.117 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtl1TH6ORAAQMEd7GQJFqk28L6HfaYPlg==", 2023-11-08T08:21:28.117 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.118 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:21:28.118 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-08T08:21:28.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:28.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:28.119 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.y", 2023-11-08T08:21:28.119 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQAuREtlXRIMNBAAC9TkKhc8QzTE+DK2eDs/bg==", 2023-11-08T08:21:28.119 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-08T08:21:28.119 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-08T08:21:28.119 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-08T08:21:28.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-08T08:21:28.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:28.120 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-11-08T08:21:28.120 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f38480682f0 msgr2=0x7f384806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f38480682f0 0x7f384806a7a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3858000c00 tx=0x7f3858008030 comp rx=0 tx=0).stop 2023-11-08T08:21:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 msgr2=0x7f386814f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 0x7f386814f9d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3860007e60 tx=0x7f386000e3c0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 shutdown_connections 2023-11-08T08:21:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3868070d90 0x7f386814f9d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f38480682f0 0x7f384806a7a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f386806a270 0x7f3868148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 --2- 172.21.15.116:0/193725134 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f386806da40 0x7f38681485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 >> 172.21.15.116:0/193725134 conn(0x7f386806c010 msgr2=0x7f386815d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 shutdown_connections 2023-11-08T08:21:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.083+0000 7f386e0ef700 1 -- 172.21.15.116:0/193725134 wait complete. 2023-11-08T08:21:28.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.243+0000 7f136ebd0700 1 Processor -- start 2023-11-08T08:21:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.243+0000 7f136ebd0700 1 -- start start 2023-11-08T08:21:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136ebd0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f136806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136ebd0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f136815cc80 0x7f136806de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136ebd0700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136806ab80 con 0x7f136806e3a0 2023-11-08T08:21:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136ebd0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13680b2490 con 0x7f136815cc80 2023-11-08T08:21:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136ebd0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13680b1810 con 0x7f136806a270 2023-11-08T08:21:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f1367fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f136815cc80 0x7f136806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136d16d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f136806e3a0 0x7f136806ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47742/0 (socket says 172.21.15.116:47742) 2023-11-08T08:21:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136d16d700 1 -- 172.21.15.116:0/3668792178 learned_addr learned my addr 172.21.15.116:0/3668792178 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136c96c700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f136806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f1367fff700 1 -- 172.21.15.116:0/3668792178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 msgr2=0x7f136806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f1367fff700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f136806a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f1367fff700 1 -- 172.21.15.116:0/3668792178 >> v1:172.21.15.116:6789/0 conn(0x7f136806e3a0 legacy=0x7f136806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f1367fff700 1 -- 172.21.15.116:0/3668792178 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13680b1af0 con 0x7f136815cc80 2023-11-08T08:21:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f13677fe700 1 -- 172.21.15.116:0/3668792178 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046390639 0 0) 0x7f136806ab80 con 0x7f136806e3a0 2023-11-08T08:21:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.244+0000 7f136c96c700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f136806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f1367fff700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f136806de60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1354009840 tx=0x7f1354009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62723339653aa6b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f13677fe700 1 -- 172.21.15.116:0/3668792178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011830 con 0x7f136815cc80 2023-11-08T08:21:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f13677fe700 1 -- 172.21.15.116:0/3668792178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354011990 con 0x7f136815cc80 2023-11-08T08:21:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f13677fe700 1 -- 172.21.15.116:0/3668792178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011d20 con 0x7f136815cc80 2023-11-08T08:21:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 -- 172.21.15.116:0/3668792178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 msgr2=0x7f136806de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f136806de60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1354009840 tx=0x7f1354009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 -- 172.21.15.116:0/3668792178 shutdown_connections 2023-11-08T08:21:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f136806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 --2- 172.21.15.116:0/3668792178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f136806de60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 -- 172.21.15.116:0/3668792178 >> 172.21.15.116:0/3668792178 conn(0x7f136806c010 msgr2=0x7f13680cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 -- 172.21.15.116:0/3668792178 shutdown_connections 2023-11-08T08:21:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.245+0000 7f136ebd0700 1 -- 172.21.15.116:0/3668792178 wait complete. 2023-11-08T08:21:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.247+0000 7f136ebd0700 1 Processor -- start 2023-11-08T08:21:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.247+0000 7f136ebd0700 1 -- start start 2023-11-08T08:21:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136ebd0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f1368153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136ebd0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f136806e3a0 0x7f13680ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136ebd0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f13680ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136ebd0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136815e880 con 0x7f136815cc80 2023-11-08T08:21:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136ebd0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136815dc20 con 0x7f136806e3a0 2023-11-08T08:21:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136ebd0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136815dee0 con 0x7f136806a270 2023-11-08T08:21:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f13680ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f13680ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47944/0 (socket says 172.21.15.116:47944) 2023-11-08T08:21:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 -- 172.21.15.116:0/1533590427 learned_addr learned my addr 172.21.15.116:0/1533590427 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f1367fff700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f136806e3a0 0x7f13680ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 -- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 msgr2=0x7f1368153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136c96c700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f1368153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f1368153130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 -- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f136806e3a0 msgr2=0x7f13680ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f136806e3a0 0x7f13680ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f136d16d700 1 -- 172.21.15.116:0/1533590427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540094a0 con 0x7f136815cc80 2023-11-08T08:21:28.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.248+0000 7f1367fff700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f136806e3a0 0x7f13680ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:28.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.249+0000 7f136d16d700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f13680ade80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f136000cba0 tx=0x7f136000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.249+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136000ab80 con 0x7f136815cc80 2023-11-08T08:21:28.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.249+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136000ad00 con 0x7f136815cc80 2023-11-08T08:21:28.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.249+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360007310 con 0x7f136815cc80 2023-11-08T08:21:28.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.249+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13680b20b0 con 0x7f136815cc80 2023-11-08T08:21:28.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.249+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13680b1dd0 con 0x7f136815cc80 2023-11-08T08:21:28.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.250+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136806e7e0 con 0x7f136815cc80 2023-11-08T08:21:28.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.250+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1360010070 con 0x7f136815cc80 2023-11-08T08:21:28.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.250+0000 7f13657fa700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f134c068320 0x7f134c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.251+0000 7f136c96c700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f134c068320 0x7f134c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.251+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f13600852a0 con 0x7f136815cc80 2023-11-08T08:21:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.251+0000 7f136c96c700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f134c068320 0x7f134c06a7d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f135c006fd0 tx=0x7f135c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.255+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1360051df0 con 0x7f136815cc80 2023-11-08T08:21:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.437+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f136815e9c0 con 0x7f136815cc80 2023-11-08T08:21:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.438+0000 7f13657fa700 1 -- 172.21.15.116:0/1533590427 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+14257 (secure 0 0 0) 0x7f136815e9c0 con 0x7f136815cc80 2023-11-08T08:21:28.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:28.439 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:27.666187+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f134c068320 msgr2=0x7f134c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f134c068320 0x7f134c06a7d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f135c006fd0 tx=0x7f135c008040 comp rx=0 tx=0).stop 2023-11-08T08:21:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 msgr2=0x7f13680ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f13680ade80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f136000cba0 tx=0x7f136000cf60 comp rx=0 tx=0).stop 2023-11-08T08:21:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 shutdown_connections 2023-11-08T08:21:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f136815cc80 0x7f13680ade80 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f136806a270 0x7f1368153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f134c068320 0x7f134c06a7d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 --2- 172.21.15.116:0/1533590427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f136806e3a0 0x7f13680ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 >> 172.21.15.116:0/1533590427 conn(0x7f136806c010 msgr2=0x7f1368159f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:28.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 shutdown_connections 2023-11-08T08:21:28.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.440+0000 7f136ebd0700 1 -- 172.21.15.116:0/1533590427 wait complete. 2023-11-08T08:21:28.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:21:28.454 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:21:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.576+0000 7fe572412700 1 Processor -- start 2023-11-08T08:21:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.577+0000 7fe572412700 1 -- start start 2023-11-08T08:21:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.577+0000 7fe572412700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.577+0000 7fe572412700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe56c15d880 0x7fe56c0cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.577+0000 7fe572412700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c15ceb0 con 0x7fe56c0cec00 2023-11-08T08:21:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.577+0000 7fe572412700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c1608c0 con 0x7fe56c15d880 2023-11-08T08:21:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.577+0000 7fe572412700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c160ba0 con 0x7fe56c160300 2023-11-08T08:21:28.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe56b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe56c15d880 0x7fe56c0cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe56c15d880 0x7fe56c0cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:39984/0 (socket says 172.21.15.116:39984) 2023-11-08T08:21:28.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 -- 172.21.15.116:0/865143298 learned_addr learned my addr 172.21.15.116:0/865143298 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe56bfff700 1 --1- 172.21.15.116:0/865143298 >> v1:172.21.15.129:6789/0 conn(0x7fe56c0cec00 0x7fe56c065fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57052/0 (socket says 172.21.15.116:57052) 2023-11-08T08:21:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 -- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 msgr2=0x7fe56c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 --2- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c15d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 -- 172.21.15.116:0/865143298 >> v1:172.21.15.129:6789/0 conn(0x7fe56c0cec00 legacy=0x7fe56c065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 -- 172.21.15.116:0/865143298 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c160e80 con 0x7fe56c15d880 2023-11-08T08:21:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe56b7fe700 1 --2- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe5709af700 1 --2- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c15d880 0x7fe56c0cac90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe564009e50 tx=0x7fe56400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82eb16f3a4b58eb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe56affd700 1 -- 172.21.15.116:0/865143298 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400e070 con 0x7fe56c15d880 2023-11-08T08:21:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe56affd700 1 -- 172.21.15.116:0/865143298 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56400b910 con 0x7fe56c15d880 2023-11-08T08:21:28.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.578+0000 7fe56affd700 1 -- 172.21.15.116:0/865143298 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400bcc0 con 0x7fe56c15d880 2023-11-08T08:21:28.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 -- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c15d880 msgr2=0x7fe56c0cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 --2- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c15d880 0x7fe56c0cac90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe564009e50 tx=0x7fe56400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:28.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 -- 172.21.15.116:0/865143298 shutdown_connections 2023-11-08T08:21:28.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 --2- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c15d880 0x7fe56c0cac90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 --2- 172.21.15.116:0/865143298 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 -- 172.21.15.116:0/865143298 >> 172.21.15.116:0/865143298 conn(0x7fe56c0cf400 msgr2=0x7fe56c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:28.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 -- 172.21.15.116:0/865143298 shutdown_connections 2023-11-08T08:21:28.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.579+0000 7fe572412700 1 -- 172.21.15.116:0/865143298 wait complete. 2023-11-08T08:21:28.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.580+0000 7fe572412700 1 Processor -- start 2023-11-08T08:21:28.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.580+0000 7fe572412700 1 -- start start 2023-11-08T08:21:28.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe572412700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c0cec00 0x7fe56c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe572412700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 0x7fe56c0c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe572412700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe572412700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c15e8f0 con 0x7fe56c15d880 2023-11-08T08:21:28.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe572412700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c15ebd0 con 0x7fe56c0cec00 2023-11-08T08:21:28.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe572412700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c15eeb0 con 0x7fe56c160300 2023-11-08T08:21:28.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe56b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 0x7fe56c0c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe56b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 0x7fe56c0c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47948/0 (socket says 172.21.15.116:47948) 2023-11-08T08:21:28.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe56b7fe700 1 -- 172.21.15.116:0/4072770845 learned_addr learned my addr 172.21.15.116:0/4072770845 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:28.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe56bfff700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c0cec00 0x7fe56c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe5709af700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe56b7fe700 1 -- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 msgr2=0x7fe56c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.581+0000 7fe56b7fe700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe56b7fe700 1 -- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c0cec00 msgr2=0x7fe56c0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe56b7fe700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c0cec00 0x7fe56c0b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe56b7fe700 1 -- 172.21.15.116:0/4072770845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c0094a0 con 0x7fe56c15d880 2023-11-08T08:21:28.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe5709af700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:28.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe56b7fe700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 0x7fe56c0c8d40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00ebe0 tx=0x7fe55c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00c910 con 0x7fe56c15d880 2023-11-08T08:21:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c00ca70 con 0x7fe56c15d880 2023-11-08T08:21:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe564009b30 con 0x7fe56c15d880 2023-11-08T08:21:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00cde0 con 0x7fe56c15d880 2023-11-08T08:21:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.582+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c0cefd0 con 0x7fe56c15d880 2023-11-08T08:21:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.583+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c161160 con 0x7fe56c15d880 2023-11-08T08:21:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.583+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe55c019030 con 0x7fe56c15d880 2023-11-08T08:21:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.583+0000 7fe568ff9700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe558068320 0x7fe55806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.584+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe55c085a60 con 0x7fe56c15d880 2023-11-08T08:21:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.584+0000 7fe56bfff700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe558068320 0x7fe55806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.584+0000 7fe56bfff700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe558068320 0x7fe55806a7d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe560000f60 tx=0x7fe560007d10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.588+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe55c052580 con 0x7fe56c15d880 2023-11-08T08:21:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.770+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe56c0cf160 con 0x7fe56c15d880 2023-11-08T08:21:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.770+0000 7fe568ff9700 1 -- 172.21.15.116:0/4072770845 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v52) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe55c0170a0 con 0x7fe56c15d880 2023-11-08T08:21:28.772 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:21:28.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe558068320 msgr2=0x7fe55806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe558068320 0x7fe55806a7d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe560000f60 tx=0x7fe560007d10 comp rx=0 tx=0).stop 2023-11-08T08:21:28.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 msgr2=0x7fe56c0c8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 0x7fe56c0c8d40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00ebe0 tx=0x7fe55c009ec0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 shutdown_connections 2023-11-08T08:21:28.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe56c160300 0x7fe56c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe558068320 0x7fe55806a7d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe56c0cec00 0x7fe56c0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 --2- 172.21.15.116:0/4072770845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c15d880 0x7fe56c0c8d40 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 >> 172.21.15.116:0/4072770845 conn(0x7fe56c0cf400 msgr2=0x7fe56c0ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:28.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 shutdown_connections 2023-11-08T08:21:28.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.772+0000 7fe572412700 1 -- 172.21.15.116:0/4072770845 wait complete. 2023-11-08T08:21:28.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:21:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.906+0000 7fe023c94700 1 Processor -- start 2023-11-08T08:21:28.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.907+0000 7fe023c94700 1 -- start start 2023-11-08T08:21:28.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.907+0000 7fe023c94700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe01c15fe50 0x7fe01c06d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.907+0000 7fe023c94700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c161370 con 0x7fe01c0b3b80 2023-11-08T08:21:28.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.907+0000 7fe023c94700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c0cf700 con 0x7fe01c0b27d0 2023-11-08T08:21:28.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.907+0000 7fe023c94700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c160220 con 0x7fe01c15fe50 2023-11-08T08:21:28.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe02122f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe01c0b3b80 0x7fe01c160ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47752/0 (socket says 172.21.15.116:47752) 2023-11-08T08:21:28.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe02122f700 1 -- 172.21.15.116:0/3433793247 learned_addr learned my addr 172.21.15.116:0/3433793247 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:28.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828183906 0 0) 0x7fe01c161370 con 0x7fe01c0b3b80 2023-11-08T08:21:28.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe021a30700 1 --2- 172.21.15.116:0/3433793247 >> v2:172.21.15.129:3300/0 conn(0x7fe01c15fe50 0x7fe01c06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0080035b0 con 0x7fe01c0b3b80 2023-11-08T08:21:28.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241199815 0 0) 0x7fe0080035b0 con 0x7fe01c0b3b80 2023-11-08T08:21:28.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 >> v2:172.21.15.129:3300/0 conn(0x7fe01c15fe50 msgr2=0x7fe01c06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:28.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 --2- 172.21.15.116:0/3433793247 >> v2:172.21.15.129:3300/0 conn(0x7fe01c15fe50 0x7fe01c06d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 >> v1:172.21.15.129:6789/0 conn(0x7fe01c0b27d0 legacy=0x7fe01c0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c0b3fd0 con 0x7fe01c0b3b80 2023-11-08T08:21:28.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fe00c001670 con 0x7fe01c0b3b80 2023-11-08T08:21:28.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 >> v1:172.21.15.116:6789/0 conn(0x7fe01c0b3b80 legacy=0x7fe01c160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe008003a00 0x7fe008003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe008004850 0x7fe008004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008005200 0x7fe00800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c160220 con 0x7fe008004850 2023-11-08T08:21:28.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00800c080 con 0x7fe008003a00 2023-11-08T08:21:28.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00800c240 con 0x7fe008005200 2023-11-08T08:21:28.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe02122f700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe008004850 0x7fe008004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe021a30700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe008003a00 0x7fe008003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.908+0000 7fe022231700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008005200 0x7fe00800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe02122f700 1 -- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008005200 msgr2=0x7fe00800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:28.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe02122f700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008005200 0x7fe00800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe02122f700 1 -- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe008003a00 msgr2=0x7fe008003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:28.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe02122f700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe008003a00 0x7fe008003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe022231700 1 -- 172.21.15.116:0/3433793247 reap_dead start 2023-11-08T08:21:28.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe02122f700 1 -- 172.21.15.116:0/3433793247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0080035b0 con 0x7fe008004850 2023-11-08T08:21:28.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe02122f700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe008004850 0x7fe008004cc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c240 tx=0x7fe008005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a3ade40f4d9e5b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c007c10 con 0x7fe008004850 2023-11-08T08:21:28.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c00cde0 con 0x7fe008004850 2023-11-08T08:21:28.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe020a2e700 1 -- 172.21.15.116:0/3433793247 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c005ba0 con 0x7fe008004850 2023-11-08T08:21:28.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 -- 172.21.15.116:0/3433793247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe008004850 msgr2=0x7fe008004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe008004850 0x7fe008004cc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c240 tx=0x7fe008005880 comp rx=0 tx=0).stop 2023-11-08T08:21:28.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 -- 172.21.15.116:0/3433793247 shutdown_connections 2023-11-08T08:21:28.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 --2- 172.21.15.116:0/3433793247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe008004850 0x7fe008004cc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 -- 172.21.15.116:0/3433793247 >> 172.21.15.116:0/3433793247 conn(0x7fe01c06ea80 msgr2=0x7fe01c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:28.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 -- 172.21.15.116:0/3433793247 shutdown_connections 2023-11-08T08:21:28.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.909+0000 7fe023c94700 1 -- 172.21.15.116:0/3433793247 wait complete. 2023-11-08T08:21:28.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 Processor -- start 2023-11-08T08:21:28.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 -- start start 2023-11-08T08:21:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008004850 0x7fe01c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 0x7fe01c0cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01c0ccf90 0x7fe01c0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c157910 con 0x7fe01c157b50 2023-11-08T08:21:28.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0ad8c0 con 0x7fe01c0ccf90 2023-11-08T08:21:28.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe023c94700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0cb870 con 0x7fe008004850 2023-11-08T08:21:28.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.911+0000 7fe02122f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 0x7fe01c0cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 0x7fe01c0cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47958/0 (socket says 172.21.15.116:47958) 2023-11-08T08:21:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 -- 172.21.15.116:0/3186325317 learned_addr learned my addr 172.21.15.116:0/3186325317 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe022231700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01c0ccf90 0x7fe01c0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe021a30700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008004850 0x7fe01c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 -- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008004850 msgr2=0x7fe01c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008004850 0x7fe01c0cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 -- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01c0ccf90 msgr2=0x7fe01c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01c0ccf90 0x7fe01c0cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 -- 172.21.15.116:0/3186325317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c0cf0b0 con 0x7fe01c157b50 2023-11-08T08:21:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe02122f700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 0x7fe01c0cca50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c050 tx=0x7fe00c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c000ed0 con 0x7fe01c157b50 2023-11-08T08:21:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c0070d0 con 0x7fe01c157b50 2023-11-08T08:21:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c0cedd0 con 0x7fe01c157b50 2023-11-08T08:21:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c007480 con 0x7fe01c157b50 2023-11-08T08:21:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.912+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c0cf390 con 0x7fe01c157b50 2023-11-08T08:21:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.913+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c0ada00 con 0x7fe01c157b50 2023-11-08T08:21:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.913+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe00c024070 con 0x7fe01c157b50 2023-11-08T08:21:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.914+0000 7fe0127fc700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0000682f0 0x7fe00006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.914+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe00c086720 con 0x7fe01c157b50 2023-11-08T08:21:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.914+0000 7fe021a30700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0000682f0 0x7fe00006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.914+0000 7fe021a30700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0000682f0 0x7fe00006a7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe018009ce0 tx=0x7fe018009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:28.918+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe01c0ada00 con 0x7fe01c157b50 2023-11-08T08:21:29.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.099+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe01c0ada00 con 0x7fe01c157b50 2023-11-08T08:21:29.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.099+0000 7fe0127fc700 1 -- 172.21.15.116:0/3186325317 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe01c0ada00 con 0x7fe01c157b50 2023-11-08T08:21:29.102 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:21:29.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0000682f0 msgr2=0x7fe00006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0000682f0 0x7fe00006a7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe018009ce0 tx=0x7fe018009210 comp rx=0 tx=0).stop 2023-11-08T08:21:29.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 msgr2=0x7fe01c0cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 0x7fe01c0cca50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c050 tx=0x7fe00c001a00 comp rx=0 tx=0).stop 2023-11-08T08:21:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 shutdown_connections 2023-11-08T08:21:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe0000682f0 0x7fe00006a7a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01c0ccf90 0x7fe01c0cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe008004850 0x7fe01c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 --2- 172.21.15.116:0/3186325317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe01c157b50 0x7fe01c0cca50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 >> 172.21.15.116:0/3186325317 conn(0x7fe01c06ea80 msgr2=0x7fe01c0b5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 shutdown_connections 2023-11-08T08:21:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.101+0000 7fe023c94700 1 -- 172.21.15.116:0/3186325317 wait complete. 2023-11-08T08:21:29.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-08T08:21:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.233+0000 7fe6dd6c6700 1 Processor -- start 2023-11-08T08:21:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.233+0000 7fe6dd6c6700 1 -- start start 2023-11-08T08:21:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6dd6c6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 0x7fe6d813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6dd6c6700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d815e010 con 0x7fe6d80ce040 2023-11-08T08:21:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6dd6c6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d813fc90 con 0x7fe6d80b1770 2023-11-08T08:21:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6dd6c6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d813f6d0 con 0x7fe6d8142db0 2023-11-08T08:21:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d6ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe6d80ce040 0x7fe6d815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47756/0 (socket says 172.21.15.116:47756) 2023-11-08T08:21:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d6ffd700 1 -- 172.21.15.116:0/1369962542 learned_addr learned my addr 172.21.15.116:0/1369962542 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323265750 0 0) 0x7fe6c8001110 con 0x7fe6d80ce040 2023-11-08T08:21:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d67fc700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 0x7fe6d813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c00035b0 con 0x7fe6d80ce040 2023-11-08T08:21:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639234786 0 0) 0x7fe6c00035b0 con 0x7fe6d80ce040 2023-11-08T08:21:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 msgr2=0x7fe6d813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 0x7fe6d813ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 >> v1:172.21.15.129:6789/0 conn(0x7fe6d80b1770 legacy=0x7fe6d813cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d813f9b0 con 0x7fe6d80ce040 2023-11-08T08:21:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fe6c8001800 con 0x7fe6d80ce040 2023-11-08T08:21:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 >> v1:172.21.15.116:6789/0 conn(0x7fe6d80ce040 legacy=0x7fe6d815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6c0003a30 0x7fe6c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 0x7fe6c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0004d90 0x7fe6c0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d813f6d0 con 0x7fe6c0004d90 2023-11-08T08:21:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c000ba10 con 0x7fe6c0003a30 2023-11-08T08:21:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c000bba0 con 0x7fe6d8142db0 2023-11-08T08:21:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d77fe700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0004d90 0x7fe6c0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.234+0000 7fe6d6ffd700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6c0003a30 0x7fe6c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d67fc700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 0x7fe6c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 -- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 msgr2=0x7fe6c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6d8142db0 0x7fe6c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 -- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6c0003a30 msgr2=0x7fe6c0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6c0003a30 0x7fe6c0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 -- 172.21.15.116:0/1369962542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00035b0 con 0x7fe6c0004d90 2023-11-08T08:21:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0004d90 0x7fe6c0005200 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c290 tx=0x7fe6c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8dc52662e111ea server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0001e70 con 0x7fe6c0004d90 2023-11-08T08:21:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d000ce50 con 0x7fe6c0004d90 2023-11-08T08:21:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d5ffb700 1 -- 172.21.15.116:0/1369962542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0012660 con 0x7fe6c0004d90 2023-11-08T08:21:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/1369962542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0004d90 msgr2=0x7fe6c0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/1369962542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0004d90 0x7fe6c0005200 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c290 tx=0x7fe6c0005880 comp rx=0 tx=0).stop 2023-11-08T08:21:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.235+0000 7fe6d77fe700 1 -- 172.21.15.116:0/1369962542 reap_dead start 2023-11-08T08:21:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.236+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/1369962542 shutdown_connections 2023-11-08T08:21:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.236+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/1369962542 >> 172.21.15.116:0/1369962542 conn(0x7fe6d806c010 msgr2=0x7fe6d80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.236+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/1369962542 shutdown_connections 2023-11-08T08:21:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.236+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/1369962542 wait complete. 2023-11-08T08:21:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.237+0000 7fe6dd6c6700 1 Processor -- start 2023-11-08T08:21:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.237+0000 7fe6dd6c6700 1 -- start start 2023-11-08T08:21:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6dd6c6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6c0004d90 0x7fe6d80c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6dd6c6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 0x7fe6d80cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6dd6c6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d8155750 0x7fe6d8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6dd6c6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80b35d0 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6dd6c6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80b38b0 con 0x7fe6d8155750 2023-11-08T08:21:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6dd6c6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80b3b90 con 0x7fe6c0004d90 2023-11-08T08:21:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6d67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 0x7fe6d80cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6d67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 0x7fe6d80cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47980/0 (socket says 172.21.15.116:47980) 2023-11-08T08:21:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6d67fc700 1 -- 172.21.15.116:0/3026673789 learned_addr learned my addr 172.21.15.116:0/3026673789 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6d6ffd700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6c0004d90 0x7fe6d80c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.238+0000 7fe6d77fe700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d8155750 0x7fe6d8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d67fc700 1 -- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6c0004d90 msgr2=0x7fe6d80c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d67fc700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6c0004d90 0x7fe6d80c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d67fc700 1 -- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d8155750 msgr2=0x7fe6d8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d67fc700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d8155750 0x7fe6d8155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d67fc700 1 -- 172.21.15.116:0/3026673789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d815d3e0 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d77fe700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d8155750 0x7fe6d8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6d67fc700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 0x7fe6d80cb950 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc011aa0 tx=0x7fe6cc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc00f980 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc00fae0 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc00fe90 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d815d100 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.239+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d80ce410 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.240+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d80743f0 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.240+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe6cc014440 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.240+0000 7fe6bf7fe700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6c40682f0 0x7fe6c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.241+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe6cc086220 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.241+0000 7fe6d6ffd700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6c40682f0 0x7fe6c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.241+0000 7fe6d6ffd700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6c40682f0 0x7fe6c406a7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8001050 tx=0x7fe6c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.245+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6cc052e30 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.427+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe6d813f1b0 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.427+0000 7fe6bf7fe700 1 -- 172.21.15.116:0/3026673789 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) v1 ==== 84+0+126 (secure 0 0 0) 0x7fe6cc022090 con 0x7fe6d80cb4e0 2023-11-08T08:21:29.429 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:29.429 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-08T08:21:29.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:29.430 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-08T08:21:29.430 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": "rbd" 2023-11-08T08:21:29.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:29.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:29.430 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 2, 2023-11-08T08:21:29.431 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-08T08:21:29.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:29.431 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-08T08:21:29.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.429+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6c40682f0 msgr2=0x7fe6c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.429+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6c40682f0 0x7fe6c406a7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8001050 tx=0x7fe6c8009040 comp rx=0 tx=0).stop 2023-11-08T08:21:29.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.429+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 msgr2=0x7fe6d80cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.429+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 0x7fe6d80cb950 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc011aa0 tx=0x7fe6cc011e60 comp rx=0 tx=0).stop 2023-11-08T08:21:29.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 shutdown_connections 2023-11-08T08:21:29.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe6c40682f0 0x7fe6c406a7a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d8155750 0x7fe6d8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe6c0004d90 0x7fe6d80c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 --2- 172.21.15.116:0/3026673789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d80cb4e0 0x7fe6d80cb950 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 >> 172.21.15.116:0/3026673789 conn(0x7fe6d806c010 msgr2=0x7fe6d8070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:29.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 shutdown_connections 2023-11-08T08:21:29.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.430+0000 7fe6dd6c6700 1 -- 172.21.15.116:0/3026673789 wait complete. 2023-11-08T08:21:29.443 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-08T08:21:29.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-08T08:21:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.567+0000 7fd45c23e700 1 Processor -- start 2023-11-08T08:21:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.567+0000 7fd45c23e700 1 -- start start 2023-11-08T08:21:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd45c23e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd45415b090 0x7fd45415b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd45c23e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd4540c9bd0 0x7fd45406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd45c23e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd45406c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd45c23e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45415b9a0 con 0x7fd45415b090 2023-11-08T08:21:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd45c23e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45406cd50 con 0x7fd4540c9bd0 2023-11-08T08:21:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd45c23e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd454071500 con 0x7fd45406c390 2023-11-08T08:21:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd459fda700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd45415b090 0x7fd45415b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd459fda700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd45415b090 0x7fd45415b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47992/0 (socket says 172.21.15.116:47992) 2023-11-08T08:21:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.568+0000 7fd459fda700 1 -- 172.21.15.116:0/3730865692 learned_addr learned my addr 172.21.15.116:0/3730865692 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd4597d9700 1 --2- 172.21.15.116:0/3730865692 >> v2:172.21.15.129:3300/0 conn(0x7fd4540c9bd0 0x7fd45406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd459fda700 1 -- 172.21.15.116:0/3730865692 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 msgr2=0x7fd45406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd45a7db700 1 --2- 172.21.15.116:0/3730865692 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd45406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd459fda700 1 --2- 172.21.15.116:0/3730865692 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd45406c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd459fda700 1 -- 172.21.15.116:0/3730865692 >> v2:172.21.15.129:3300/0 conn(0x7fd4540c9bd0 msgr2=0x7fd45406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd459fda700 1 --2- 172.21.15.116:0/3730865692 >> v2:172.21.15.129:3300/0 conn(0x7fd4540c9bd0 0x7fd45406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd459fda700 1 -- 172.21.15.116:0/3730865692 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45406aaa0 con 0x7fd45415b090 2023-11-08T08:21:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd4597d9700 1 --2- 172.21.15.116:0/3730865692 >> v2:172.21.15.129:3300/0 conn(0x7fd4540c9bd0 0x7fd45406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd459fda700 1 --2- 172.21.15.116:0/3730865692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45415b090 0x7fd45415b460 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd4540c2e70 tx=0x7fd444009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f7e1789c0a8eca server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd458fd8700 1 -- 172.21.15.116:0/3730865692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44400b870 con 0x7fd45415b090 2023-11-08T08:21:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd458fd8700 1 -- 172.21.15.116:0/3730865692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44400b9d0 con 0x7fd45415b090 2023-11-08T08:21:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.569+0000 7fd458fd8700 1 -- 172.21.15.116:0/3730865692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44400bd60 con 0x7fd45415b090 2023-11-08T08:21:29.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 -- 172.21.15.116:0/3730865692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45415b090 msgr2=0x7fd45415b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 --2- 172.21.15.116:0/3730865692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45415b090 0x7fd45415b460 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd4540c2e70 tx=0x7fd444009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 -- 172.21.15.116:0/3730865692 shutdown_connections 2023-11-08T08:21:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 --2- 172.21.15.116:0/3730865692 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd45406c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 --2- 172.21.15.116:0/3730865692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45415b090 0x7fd45415b460 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 --2- 172.21.15.116:0/3730865692 >> v2:172.21.15.129:3300/0 conn(0x7fd4540c9bd0 0x7fd45406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 -- 172.21.15.116:0/3730865692 >> 172.21.15.116:0/3730865692 conn(0x7fd4540cf370 msgr2=0x7fd4540cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 -- 172.21.15.116:0/3730865692 shutdown_connections 2023-11-08T08:21:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.570+0000 7fd45c23e700 1 -- 172.21.15.116:0/3730865692 wait complete. 2023-11-08T08:21:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.571+0000 7fd45c23e700 1 Processor -- start 2023-11-08T08:21:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.571+0000 7fd45c23e700 1 -- start start 2023-11-08T08:21:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd454153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 0x7fd4540b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd45415b090 0x7fd4541528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd454152df0 con 0x7fd4540c9bd0 2023-11-08T08:21:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540c7f40 con 0x7fd45415b090 2023-11-08T08:21:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540c8220 con 0x7fd45406c390 2023-11-08T08:21:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 0x7fd4540b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 0x7fd4540b4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47996/0 (socket says 172.21.15.116:47996) 2023-11-08T08:21:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 -- 172.21.15.116:0/387342582 learned_addr learned my addr 172.21.15.116:0/387342582 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd459fda700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd454153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45a7db700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd45415b090 0x7fd4541528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 -- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 msgr2=0x7fd454153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd454153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 -- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd45415b090 msgr2=0x7fd4541528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd45415b090 0x7fd4541528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 -- 172.21.15.116:0/387342582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd444009550 con 0x7fd4540c9bd0 2023-11-08T08:21:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd4597d9700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 0x7fd4540b4e80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd45000c750 tx=0x7fd45000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd450011800 con 0x7fd4540c9bd0 2023-11-08T08:21:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd450011960 con 0x7fd4540c9bd0 2023-11-08T08:21:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45406b060 con 0x7fd4540c9bd0 2023-11-08T08:21:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd450011cd0 con 0x7fd4540c9bd0 2023-11-08T08:21:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.572+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45406ad80 con 0x7fd4540c9bd0 2023-11-08T08:21:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.573+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45406b340 con 0x7fd4540c9bd0 2023-11-08T08:21:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.574+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd450020070 con 0x7fd4540c9bd0 2023-11-08T08:21:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.574+0000 7fd44a7fc700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd4380682f0 0x7fd43806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.574+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd4500857c0 con 0x7fd4540c9bd0 2023-11-08T08:21:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.574+0000 7fd459fda700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd4380682f0 0x7fd43806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.575+0000 7fd459fda700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd4380682f0 0x7fd43806a7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd444000c00 tx=0x7fd444008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.578+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd450052370 con 0x7fd4540c9bd0 2023-11-08T08:21:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:29.761+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd4540743f0 con 0x7fd4540c9bd0 2023-11-08T08:21:30.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.682+0000 7fd44a7fc700 1 -- 172.21.15.116:0/387342582 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd450055990 con 0x7fd4540c9bd0 2023-11-08T08:21:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd4380682f0 msgr2=0x7fd43806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd4380682f0 0x7fd43806a7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd444000c00 tx=0x7fd444008030 comp rx=0 tx=0).stop 2023-11-08T08:21:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 msgr2=0x7fd4540b4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 0x7fd4540b4e80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd45000c750 tx=0x7fd45000cb10 comp rx=0 tx=0).stop 2023-11-08T08:21:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 shutdown_connections 2023-11-08T08:21:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd4380682f0 0x7fd43806a7a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd45415b090 0x7fd4541528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd45406c390 0x7fd454153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 --2- 172.21.15.116:0/387342582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4540c9bd0 0x7fd4540b4e80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.686+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 >> 172.21.15.116:0/387342582 conn(0x7fd4540cf370 msgr2=0x7fd4540cf740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.687+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 shutdown_connections 2023-11-08T08:21:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.687+0000 7fd45c23e700 1 -- 172.21.15.116:0/387342582 wait complete. 2023-11-08T08:21:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-08T08:21:30.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-08T08:21:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.835+0000 7f3a731df700 1 Processor -- start 2023-11-08T08:21:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.835+0000 7f3a731df700 1 -- start start 2023-11-08T08:21:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.835+0000 7f3a731df700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3a6c0cec00 0x7f3a6c065fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.835+0000 7f3a731df700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c15ceb0 con 0x7f3a6c160300 2023-11-08T08:21:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.835+0000 7f3a731df700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c1608c0 con 0x7f3a6c0cac90 2023-11-08T08:21:30.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.835+0000 7f3a731df700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c160ba0 con 0x7f3a6c0cec00 2023-11-08T08:21:30.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6bfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3a6c160300 0x7f3a6c15d340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47760/0 (socket says 172.21.15.116:47760) 2023-11-08T08:21:30.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6bfff700 1 -- 172.21.15.116:0/4045598156 learned_addr learned my addr 172.21.15.116:0/4045598156 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:30.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/4045598156 >> v2:172.21.15.129:3300/0 conn(0x7f3a6c0cec00 0x7f3a6c065fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164143221 0 0) 0x7f3a6c15ceb0 con 0x7f3a6c160300 2023-11-08T08:21:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a5c0035b0 con 0x7f3a6c160300 2023-11-08T08:21:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811229837 0 0) 0x7f3a6c1608c0 con 0x7f3a6c0cac90 2023-11-08T08:21:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6c15ceb0 con 0x7f3a6c0cac90 2023-11-08T08:21:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1942381057 0 0) 0x7f3a5c0035b0 con 0x7f3a6c160300 2023-11-08T08:21:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 >> v2:172.21.15.129:3300/0 conn(0x7f3a6c0cec00 msgr2=0x7f3a6c065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 --2- 172.21.15.116:0/4045598156 >> v2:172.21.15.129:3300/0 conn(0x7f3a6c0cec00 0x7f3a6c065fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 >> v1:172.21.15.129:6789/0 conn(0x7f3a6c0cac90 legacy=0x7f3a6c06c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c160e80 con 0x7f3a6c160300 2023-11-08T08:21:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f3a58001670 con 0x7f3a6c160300 2023-11-08T08:21:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 >> v1:172.21.15.116:6789/0 conn(0x7f3a6c160300 legacy=0x7f3a6c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a5c003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a5c004850 0x7f3a5c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a5c005200 0x7f3a5c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c160ba0 con 0x7f3a5c003be0 2023-11-08T08:21:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c00c0e0 con 0x7f3a5c005200 2023-11-08T08:21:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c00c270 con 0x7f3a5c004850 2023-11-08T08:21:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a5c003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a7177c700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a5c005200 0x7f3a5c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.836+0000 7f3a6bfff700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a5c004850 0x7f3a5c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a70f7b700 1 -- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a5c004850 msgr2=0x7f3a5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a5c004850 0x7f3a5c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a70f7b700 1 -- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a5c005200 msgr2=0x7f3a5c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a5c005200 0x7f3a5c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a7177c700 1 -- 172.21.15.116:0/4045598156 reap_dead start 2023-11-08T08:21:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a70f7b700 1 -- 172.21.15.116:0/4045598156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c0035b0 con 0x7f3a5c003be0 2023-11-08T08:21:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a6bfff700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a5c004850 0x7f3a5c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a5c003fd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009b30 tx=0x7f3a5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d0bb8aea506849 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000e6c0 con 0x7f3a5c003be0 2023-11-08T08:21:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6000e820 con 0x7f3a5c003be0 2023-11-08T08:21:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a6b7fe700 1 -- 172.21.15.116:0/4045598156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000eb90 con 0x7f3a5c003be0 2023-11-08T08:21:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a731df700 1 -- 172.21.15.116:0/4045598156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 msgr2=0x7f3a5c003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a731df700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a5c003fd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009b30 tx=0x7f3a5c005880 comp rx=0 tx=0).stop 2023-11-08T08:21:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a731df700 1 -- 172.21.15.116:0/4045598156 shutdown_connections 2023-11-08T08:21:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a731df700 1 --2- 172.21.15.116:0/4045598156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a5c003fd0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a731df700 1 -- 172.21.15.116:0/4045598156 >> 172.21.15.116:0/4045598156 conn(0x7f3a6c0cf400 msgr2=0x7f3a6c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.837+0000 7f3a731df700 1 -- 172.21.15.116:0/4045598156 shutdown_connections 2023-11-08T08:21:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.838+0000 7f3a731df700 1 -- 172.21.15.116:0/4045598156 wait complete. 2023-11-08T08:21:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.839+0000 7f3a731df700 1 Processor -- start 2023-11-08T08:21:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.839+0000 7f3a731df700 1 -- start start 2023-11-08T08:21:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.839+0000 7f3a731df700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a6c154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a6c154200 0x7f3a6c15e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a6c0b38d0 0x7f3a6c153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0ad870 con 0x7f3a5c003be0 2023-11-08T08:21:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0adb50 con 0x7f3a6c154200 2023-11-08T08:21:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0ade30 con 0x7f3a6c0b38d0 2023-11-08T08:21:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a6c154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a6c154700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48028/0 (socket says 172.21.15.116:48028) 2023-11-08T08:21:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 -- 172.21.15.116:0/2125467899 learned_addr learned my addr 172.21.15.116:0/2125467899 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a6bfff700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a6c154200 0x7f3a6c15e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a7177c700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a6c0b38d0 0x7f3a6c153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 -- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a6c0b38d0 msgr2=0x7f3a6c153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a6c0b38d0 0x7f3a6c153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 -- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a6c154200 msgr2=0x7f3a6c15e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a6c154200 0x7f3a6c15e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 -- 172.21.15.116:0/2125467899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0cefd0 con 0x7f3a5c003be0 2023-11-08T08:21:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a70f7b700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a6c154700 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0047a0 tx=0x7f3a600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000e6c0 con 0x7f3a5c003be0 2023-11-08T08:21:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a600038e0 con 0x7f3a5c003be0 2023-11-08T08:21:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c161160 con 0x7f3a5c003be0 2023-11-08T08:21:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60003c90 con 0x7f3a5c003be0 2023-11-08T08:21:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.840+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c0cf160 con 0x7f3a5c003be0 2023-11-08T08:21:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.841+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c15def0 con 0x7f3a5c003be0 2023-11-08T08:21:30.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.842+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3a60019030 con 0x7f3a5c003be0 2023-11-08T08:21:30.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.843+0000 7f3a697fa700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3a500682f0 0x7f3a5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.843+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f3a600877f0 con 0x7f3a5c003be0 2023-11-08T08:21:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.843+0000 7f3a6bfff700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3a500682f0 0x7f3a5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.843+0000 7f3a6bfff700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3a500682f0 0x7f3a5006a7a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3a58001eb0 tx=0x7f3a58007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:30.847+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a600541c0 con 0x7f3a5c003be0 2023-11-08T08:21:31.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.027+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3a6c15def0 con 0x7f3a5c003be0 2023-11-08T08:21:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.688+0000 7f3a697fa700 1 -- 172.21.15.116:0/2125467899 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3a6c15def0 con 0x7f3a5c003be0 2023-11-08T08:21:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3a500682f0 msgr2=0x7f3a5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3a500682f0 0x7f3a5006a7a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3a58001eb0 tx=0x7f3a58007040 comp rx=0 tx=0).stop 2023-11-08T08:21:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 msgr2=0x7f3a6c154700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a6c154700 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0047a0 tx=0x7f3a600047c0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 shutdown_connections 2023-11-08T08:21:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3a6c0b38d0 0x7f3a6c153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a5c003be0 0x7f3a6c154700 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3a500682f0 0x7f3a5006a7a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 --2- 172.21.15.116:0/2125467899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a6c154200 0x7f3a6c15e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 >> 172.21.15.116:0/2125467899 conn(0x7f3a6c0cf400 msgr2=0x7f3a6c0affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.697+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 shutdown_connections 2023-11-08T08:21:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.698+0000 7f3a731df700 1 -- 172.21.15.116:0/2125467899 wait complete. 2023-11-08T08:21:31.704 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-08T08:21:31.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-08T08:21:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.847+0000 7fc77942c700 1 Processor -- start 2023-11-08T08:21:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.847+0000 7fc77942c700 1 -- start start 2023-11-08T08:21:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.848+0000 7fc77942c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc77415a5f0 0x7fc7740d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.848+0000 7fc77942c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 0x7fc7740b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.848+0000 7fc77942c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77415a9c0 con 0x7fc7740b2d10 2023-11-08T08:21:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.848+0000 7fc77942c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7740b3190 con 0x7fc77415a5f0 2023-11-08T08:21:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.848+0000 7fc77942c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740c7b80 con 0x7fc7740cc200 2023-11-08T08:21:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 0x7fc7740b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 0x7fc7740b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:58498/0 (socket says 172.21.15.116:58498) 2023-11-08T08:21:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc772ffd700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc77415a5f0 0x7fc7740d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7737fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc7740b2d10 0x7fc77415e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:57086/0 (socket says 172.21.15.116:57086) 2023-11-08T08:21:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 -- 172.21.15.116:0/3124843596 learned_addr learned my addr 172.21.15.116:0/3124843596 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 -- 172.21.15.116:0/3124843596 >> v2:172.21.15.129:3300/0 conn(0x7fc77415a5f0 msgr2=0x7fc7740d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 --2- 172.21.15.116:0/3124843596 >> v2:172.21.15.129:3300/0 conn(0x7fc77415a5f0 0x7fc7740d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 -- 172.21.15.116:0/3124843596 >> v1:172.21.15.129:6789/0 conn(0x7fc7740b2d10 legacy=0x7fc77415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 -- 172.21.15.116:0/3124843596 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7740cc990 con 0x7fc7740cc200 2023-11-08T08:21:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc772ffd700 1 --2- 172.21.15.116:0/3124843596 >> v2:172.21.15.129:3300/0 conn(0x7fc77415a5f0 0x7fc7740d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc7727fc700 1 --2- 172.21.15.116:0/3124843596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 0x7fc7740b27d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc7640098f0 tx=0x7fc764009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949852a56654f84e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc771ffb700 1 -- 172.21.15.116:0/3124843596 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764011830 con 0x7fc7740cc200 2023-11-08T08:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc771ffb700 1 -- 172.21.15.116:0/3124843596 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764011990 con 0x7fc7740cc200 2023-11-08T08:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.849+0000 7fc771ffb700 1 -- 172.21.15.116:0/3124843596 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764011d20 con 0x7fc7740cc200 2023-11-08T08:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 -- 172.21.15.116:0/3124843596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 msgr2=0x7fc7740b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 --2- 172.21.15.116:0/3124843596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 0x7fc7740b27d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc7640098f0 tx=0x7fc764009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 -- 172.21.15.116:0/3124843596 shutdown_connections 2023-11-08T08:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 --2- 172.21.15.116:0/3124843596 >> v2:172.21.15.129:3300/0 conn(0x7fc77415a5f0 0x7fc7740d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 --2- 172.21.15.116:0/3124843596 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc7740cc200 0x7fc7740b27d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 -- 172.21.15.116:0/3124843596 >> 172.21.15.116:0/3124843596 conn(0x7fc77406ea80 msgr2=0x7fc77406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 -- 172.21.15.116:0/3124843596 shutdown_connections 2023-11-08T08:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.850+0000 7fc77942c700 1 -- 172.21.15.116:0/3124843596 wait complete. 2023-11-08T08:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.851+0000 7fc77942c700 1 Processor -- start 2023-11-08T08:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.851+0000 7fc77942c700 1 -- start start 2023-11-08T08:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc77942c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 0x7fc77414a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc77942c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7740cc200 0x7fc7740b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc77942c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc77415a5f0 0x7fc7740b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc77942c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740b3b80 con 0x7fc7740b2d10 2023-11-08T08:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc77942c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77414a8c0 con 0x7fc7740cc200 2023-11-08T08:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc77942c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740ca4b0 con 0x7fc77415a5f0 2023-11-08T08:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 0x7fc77414a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 0x7fc77414a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48038/0 (socket says 172.21.15.116:48038) 2023-11-08T08:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 -- 172.21.15.116:0/2404803015 learned_addr learned my addr 172.21.15.116:0/2404803015 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc7727fc700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7740cc200 0x7fc7740b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc7737fe700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc77415a5f0 0x7fc7740b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 -- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc77415a5f0 msgr2=0x7fc7740b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc77415a5f0 0x7fc7740b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 -- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7740cc200 msgr2=0x7fc7740b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7740cc200 0x7fc7740b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.852+0000 7fc772ffd700 1 -- 172.21.15.116:0/2404803015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680094a0 con 0x7fc7740b2d10 2023-11-08T08:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.853+0000 7fc772ffd700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 0x7fc77414a380 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc76800ebe0 tx=0x7fc76800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.853+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76800c850 con 0x7fc7740b2d10 2023-11-08T08:21:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.853+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76800c9b0 con 0x7fc7740b2d10 2023-11-08T08:21:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.853+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76800cd40 con 0x7fc7740b2d10 2023-11-08T08:21:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.853+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc764009550 con 0x7fc7740b2d10 2023-11-08T08:21:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.853+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7740ccf50 con 0x7fc7740b2d10 2023-11-08T08:21:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.854+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7740ccc70 con 0x7fc7740b2d10 2023-11-08T08:21:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.854+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc768023070 con 0x7fc7740b2d10 2023-11-08T08:21:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.854+0000 7fc75b7fe700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc7600682f0 0x7fc76006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.854+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fc768085f90 con 0x7fc7740b2d10 2023-11-08T08:21:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.854+0000 7fc7727fc700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc7600682f0 0x7fc76006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.855+0000 7fc7727fc700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc7600682f0 0x7fc76006a7a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc764009c40 tx=0x7fc764008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:31.859+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7680527d0 con 0x7fc7740b2d10 2023-11-08T08:21:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.081+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc7740cd230 con 0x7fc7740b2d10 2023-11-08T08:21:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.085+0000 7fc75b7fe700 1 -- 172.21.15.116:0/2404803015 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v28) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc76800cea0 con 0x7fc7740b2d10 2023-11-08T08:21:32.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc7600682f0 msgr2=0x7fc76006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc7600682f0 0x7fc76006a7a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc764009c40 tx=0x7fc764008030 comp rx=0 tx=0).stop 2023-11-08T08:21:32.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 msgr2=0x7fc77414a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 0x7fc77414a380 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc76800ebe0 tx=0x7fc76800eef0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 shutdown_connections 2023-11-08T08:21:32.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc77415a5f0 0x7fc7740b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7740b2d10 0x7fc77414a380 secure :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc76800ebe0 tx=0x7fc76800eef0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc7600682f0 0x7fc76006a7a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 --2- 172.21.15.116:0/2404803015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7740cc200 0x7fc7740b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.090+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 >> 172.21.15.116:0/2404803015 conn(0x7fc77406ea80 msgr2=0x7fc7741590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:32.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.093+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 shutdown_connections 2023-11-08T08:21:32.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.094+0000 7fc77942c700 1 -- 172.21.15.116:0/2404803015 wait complete. 2023-11-08T08:21:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 9 and data pool 10 2023-11-08T08:21:32.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.237+0000 7fdf0160c700 1 Processor -- start 2023-11-08T08:21:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.237+0000 7fdf0160c700 1 -- start start 2023-11-08T08:21:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdf0160c700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdefc0cd5a0 0x7fdefc0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdf0160c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc0d1ee0 con 0x7fdefc0b22a0 2023-11-08T08:21:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdf0160c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc0cdb10 con 0x7fdefc06ab10 2023-11-08T08:21:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdf0160c700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdefc0d1890 con 0x7fdefc0cd5a0 2023-11-08T08:21:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdefb7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdefc0b22a0 0x7fdefc0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47762/0 (socket says 172.21.15.116:47762) 2023-11-08T08:21:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdefb7fe700 1 -- 172.21.15.116:0/294434756 learned_addr learned my addr 172.21.15.116:0/294434756 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580054387 0 0) 0x7fdefc0d1ee0 con 0x7fdefc0b22a0 2023-11-08T08:21:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee40035b0 con 0x7fdefc0b22a0 2023-11-08T08:21:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/294434756 >> v2:172.21.15.129:3300/0 conn(0x7fdefc0cd5a0 0x7fdefc0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230656136 0 0) 0x7fdee40035b0 con 0x7fdefc0b22a0 2023-11-08T08:21:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 >> v2:172.21.15.129:3300/0 conn(0x7fdefc0cd5a0 msgr2=0x7fdefc0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 --2- 172.21.15.116:0/294434756 >> v2:172.21.15.129:3300/0 conn(0x7fdefc0cd5a0 0x7fdefc0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 >> v1:172.21.15.129:6789/0 conn(0x7fdefc06ab10 legacy=0x7fdefc06aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc0d1b70 con 0x7fdefc0b22a0 2023-11-08T08:21:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fdef40019f0 con 0x7fdefc0b22a0 2023-11-08T08:21:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 >> v1:172.21.15.116:6789/0 conn(0x7fdefc0b22a0 legacy=0x7fdefc0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdee4003a00 0x7fdee4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdee4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee4005200 0x7fdee400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0d1890 con 0x7fdee4004850 2023-11-08T08:21:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee400c190 con 0x7fdee4005200 2023-11-08T08:21:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.238+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee400c350 con 0x7fdee4003a00 2023-11-08T08:21:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdee4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefb7fe700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee4005200 0x7fdee400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefaffd700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdee4003a00 0x7fdee4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 -- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdee4003a00 msgr2=0x7fdee4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdee4003a00 0x7fdee4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 -- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee4005200 msgr2=0x7fdee400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee4005200 0x7fdee400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 -- 172.21.15.116:0/294434756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee40035b0 con 0x7fdee4004850 2023-11-08T08:21:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefb7fe700 1 -- 172.21.15.116:0/294434756 reap_dead start 2023-11-08T08:21:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdee4004cc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0cda40 tx=0x7fdee4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13c8d54b7d6e182 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00c070 con 0x7fdee4004850 2023-11-08T08:21:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec004020 con 0x7fdee4004850 2023-11-08T08:21:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.239+0000 7fdef9ffb700 1 -- 172.21.15.116:0/294434756 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0043b0 con 0x7fdee4004850 2023-11-08T08:21:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 -- 172.21.15.116:0/294434756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 msgr2=0x7fdee4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdee4004cc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0cda40 tx=0x7fdee4005880 comp rx=0 tx=0).stop 2023-11-08T08:21:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 -- 172.21.15.116:0/294434756 shutdown_connections 2023-11-08T08:21:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 --2- 172.21.15.116:0/294434756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdee4004cc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 -- 172.21.15.116:0/294434756 >> 172.21.15.116:0/294434756 conn(0x7fdefc06c520 msgr2=0x7fdefc06c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 -- 172.21.15.116:0/294434756 shutdown_connections 2023-11-08T08:21:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.240+0000 7fdf0160c700 1 -- 172.21.15.116:0/294434756 wait complete. 2023-11-08T08:21:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.241+0000 7fdf0160c700 1 Processor -- start 2023-11-08T08:21:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.241+0000 7fdf0160c700 1 -- start start 2023-11-08T08:21:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdefc0caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdefc158b40 0x7fdefc1e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0b8290 0x7fdefc154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1e66b0 con 0x7fdee4004850 2023-11-08T08:21:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc158fb0 con 0x7fdefc0b8290 2023-11-08T08:21:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0cb110 con 0x7fdefc158b40 2023-11-08T08:21:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdefc0caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdefc0caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48050/0 (socket says 172.21.15.116:48050) 2023-11-08T08:21:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 -- 172.21.15.116:0/1626377371 learned_addr learned my addr 172.21.15.116:0/1626377371 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefb7fe700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0b8290 0x7fdefc154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdefc158b40 0x7fdefc1e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 -- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdefc158b40 msgr2=0x7fdefc1e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdefc158b40 0x7fdefc1e6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 -- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0b8290 msgr2=0x7fdefc154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0b8290 0x7fdefc154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 -- 172.21.15.116:0/1626377371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc0b1ec0 con 0x7fdee4004850 2023-11-08T08:21:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefb7fe700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0b8290 0x7fdefc154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdefaffd700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdefc0caa30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdef000f820 tx=0x7fdef000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef00052e0 con 0x7fdee4004850 2023-11-08T08:21:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef0005460 con 0x7fdee4004850 2023-11-08T08:21:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.242+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc15cad0 con 0x7fdee4004850 2023-11-08T08:21:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.243+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef00038c0 con 0x7fdee4004850 2023-11-08T08:21:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.243+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc0b2050 con 0x7fdee4004850 2023-11-08T08:21:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.244+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdefc0743f0 con 0x7fdee4004850 2023-11-08T08:21:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.244+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdef0003a20 con 0x7fdee4004850 2023-11-08T08:21:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.244+0000 7fdee37fe700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdee80682f0 0x7fdee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.245+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdee80682f0 0x7fdee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.245+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdef0085bc0 con 0x7fdee4004850 2023-11-08T08:21:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.245+0000 7fdefa7fc700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdee80682f0 0x7fdee806a7a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0cda40 tx=0x7fdeec00c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.249+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdef0052420 con 0x7fdee4004850 2023-11-08T08:21:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.430+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdefc0d2c80 con 0x7fdee4004850 2023-11-08T08:21:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.431+0000 7fdee37fe700 1 -- 172.21.15.116:0/1626377371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+17972 (secure 0 0 0) 0x7fdef0003cb0 con 0x7fdee4004850 2023-11-08T08:21:32.433 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:32.433 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":55,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:32.082674+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdee80682f0 msgr2=0x7fdee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdee80682f0 0x7fdee806a7a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0cda40 tx=0x7fdeec00c000 comp rx=0 tx=0).stop 2023-11-08T08:21:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 msgr2=0x7fdefc0caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdefc0caa30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdef000f820 tx=0x7fdef000fbe0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 shutdown_connections 2023-11-08T08:21:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdee80682f0 0x7fdee806a7a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0b8290 0x7fdefc154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee4004850 0x7fdefc0caa30 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 --2- 172.21.15.116:0/1626377371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdefc158b40 0x7fdefc1e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 >> 172.21.15.116:0/1626377371 conn(0x7fdefc06c520 msgr2=0x7fdefc0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 shutdown_connections 2023-11-08T08:21:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.434+0000 7fdf0160c700 1 -- 172.21.15.116:0/1626377371 wait complete. 2023-11-08T08:21:32.447 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:32.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.565+0000 7ff5d69d4700 1 Processor -- start 2023-11-08T08:21:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.565+0000 7ff5d69d4700 1 -- start start 2023-11-08T08:21:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d69d4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d69d4700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5d015e0b0 0x7ff5d015e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d69d4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d0151a50 con 0x7ff5d0070d90 2023-11-08T08:21:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d69d4700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d015b440 con 0x7ff5d015e0b0 2023-11-08T08:21:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d69d4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d006ab80 con 0x7ff5d006a270 2023-11-08T08:21:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d4f71700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5d015e0b0 0x7ff5d015e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d4f71700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5d015e0b0 0x7ff5d015e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47706/0 (socket says 172.21.15.116:47706) 2023-11-08T08:21:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.566+0000 7ff5d4f71700 1 -- 172.21.15.116:0/546349916 learned_addr learned my addr 172.21.15.116:0/546349916 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5cffff700 1 --2- 172.21.15.116:0/546349916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d4f71700 1 -- 172.21.15.116:0/546349916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 msgr2=0x7ff5d006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d4f71700 1 --2- 172.21.15.116:0/546349916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d006a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d4f71700 1 -- 172.21.15.116:0/546349916 >> v1:172.21.15.129:6789/0 conn(0x7ff5d0070d90 legacy=0x7ff5d00711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d4f71700 1 -- 172.21.15.116:0/546349916 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0071ac0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d4f71700 1 --2- 172.21.15.116:0/546349916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d015e530 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009e50 tx=0x7ff5c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72143de1d5864697 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5ceffd700 1 -- 172.21.15.116:0/546349916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c800e070 con 0x7ff5d015e0b0 2023-11-08T08:21:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5ceffd700 1 -- 172.21.15.116:0/546349916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c800b9c0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5ceffd700 1 -- 172.21.15.116:0/546349916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c800bd70 con 0x7ff5d015e0b0 2023-11-08T08:21:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d69d4700 1 -- 172.21.15.116:0/546349916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 msgr2=0x7ff5d015e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/546349916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d015e530 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009e50 tx=0x7ff5c800b330 comp rx=0 tx=0).stop 2023-11-08T08:21:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d69d4700 1 -- 172.21.15.116:0/546349916 shutdown_connections 2023-11-08T08:21:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/546349916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d015e530 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/546349916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d006a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.567+0000 7ff5d69d4700 1 -- 172.21.15.116:0/546349916 >> 172.21.15.116:0/546349916 conn(0x7ff5d006c010 msgr2=0x7ff5d00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.568+0000 7ff5d69d4700 1 -- 172.21.15.116:0/546349916 shutdown_connections 2023-11-08T08:21:32.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.568+0000 7ff5d69d4700 1 -- 172.21.15.116:0/546349916 wait complete. 2023-11-08T08:21:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.569+0000 7ff5d69d4700 1 Processor -- start 2023-11-08T08:21:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.569+0000 7ff5d69d4700 1 -- start start 2023-11-08T08:21:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d00cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d0070d90 0x7ff5d014bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00c89b0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00b3f20 con 0x7ff5d0070d90 2023-11-08T08:21:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00b4200 con 0x7ff5d006a270 2023-11-08T08:21:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d014c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47716/0 (socket says 172.21.15.116:47716) 2023-11-08T08:21:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5cffff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d00cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 -- 172.21.15.116:0/93616061 learned_addr learned my addr 172.21.15.116:0/93616061 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5cf7fe700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d0070d90 0x7ff5d014bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 -- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 msgr2=0x7ff5d00cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d00cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 -- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d0070d90 msgr2=0x7ff5d014bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d0070d90 0x7ff5d014bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 -- 172.21.15.116:0/93616061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c8009b30 con 0x7ff5d015e0b0 2023-11-08T08:21:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d4f71700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d014c490 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0067450 tx=0x7ff5c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c80047c0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c80099b0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d015b0d0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c801c680 con 0x7ff5d015e0b0 2023-11-08T08:21:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.570+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d015adf0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.571+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d015aac0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.572+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5c802c070 con 0x7ff5d015e0b0 2023-11-08T08:21:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.572+0000 7ff5ccff9700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.572+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff5c80876e0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.572+0000 7ff5cffff700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.572+0000 7ff5cffff700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4006b20 tx=0x7ff5c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.576+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5c8053f40 con 0x7ff5d015e0b0 2023-11-08T08:21:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.758+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5d00743f0 con 0x7ff5d015e0b0 2023-11-08T08:21:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.759+0000 7ff5ccff9700 1 -- 172.21.15.116:0/93616061 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+17972 (secure 0 0 0) 0x7ff5c8057560 con 0x7ff5d015e0b0 2023-11-08T08:21:32.761 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:32.761 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":55,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:32.082674+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.761+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 msgr2=0x7ff5bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.761+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4006b20 tx=0x7ff5c4008040 comp rx=0 tx=0).stop 2023-11-08T08:21:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.761+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 msgr2=0x7ff5d014c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.761+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d014c490 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0067450 tx=0x7ff5c8004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 shutdown_connections 2023-11-08T08:21:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d015e0b0 0x7ff5d014c490 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff5d006a270 0x7ff5d00cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff5bc0682f0 0x7ff5bc06a7a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 --2- 172.21.15.116:0/93616061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d0070d90 0x7ff5d014bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 >> 172.21.15.116:0/93616061 conn(0x7ff5d006c010 msgr2=0x7ff5d015baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 shutdown_connections 2023-11-08T08:21:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.762+0000 7ff5d69d4700 1 -- 172.21.15.116:0/93616061 wait complete. 2023-11-08T08:21:32.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-08T08:21:32.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.897+0000 7ff0a603f700 1 Processor -- start 2023-11-08T08:21:32.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.897+0000 7ff0a603f700 1 -- start start 2023-11-08T08:21:32.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff0a603f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff0a015a5f0 0x7ff0a00d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff0a603f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a00cc200 0x7ff0a00b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff0a603f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a015a9c0 con 0x7ff0a00b2d10 2023-11-08T08:21:32.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff0a603f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a00b3190 con 0x7ff0a015a5f0 2023-11-08T08:21:32.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff0a603f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00c7b80 con 0x7ff0a00cc200 2023-11-08T08:21:32.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09effd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a00cc200 0x7ff0a00b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09f7fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff0a015a5f0 0x7ff0a00d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09ffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff0a00b2d10 0x7ff0a015e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:38320/0 (socket says 172.21.15.116:38320) 2023-11-08T08:21:32.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09ffff700 1 -- 172.21.15.116:0/669948472 learned_addr learned my addr 172.21.15.116:0/669948472 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:32.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09f7fe700 1 -- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a00cc200 msgr2=0x7ff0a00b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:32.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09f7fe700 1 --2- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a00cc200 0x7ff0a00b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09f7fe700 1 -- 172.21.15.116:0/669948472 >> v1:172.21.15.129:6789/0 conn(0x7ff0a00b2d10 legacy=0x7ff0a015e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09f7fe700 1 -- 172.21.15.116:0/669948472 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00cc990 con 0x7ff0a015a5f0 2023-11-08T08:21:32.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.898+0000 7ff09e7fc700 1 -- 172.21.15.116:0/669948472 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100636264 0 0) 0x7ff0a015a9c0 con 0x7ff0a00b2d10 2023-11-08T08:21:32.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff09f7fe700 1 --2- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a015a5f0 0x7ff0a00d0ad0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff094009840 tx=0x7ff094009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dcc29f4967eed3a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff09e7fc700 1 -- 172.21.15.116:0/669948472 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094011830 con 0x7ff0a015a5f0 2023-11-08T08:21:32.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff09e7fc700 1 -- 172.21.15.116:0/669948472 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff094011990 con 0x7ff0a015a5f0 2023-11-08T08:21:32.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff09e7fc700 1 -- 172.21.15.116:0/669948472 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094011d20 con 0x7ff0a015a5f0 2023-11-08T08:21:32.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 -- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a015a5f0 msgr2=0x7ff0a00d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 --2- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a015a5f0 0x7ff0a00d0ad0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff094009840 tx=0x7ff094009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:32.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 -- 172.21.15.116:0/669948472 shutdown_connections 2023-11-08T08:21:32.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 --2- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a015a5f0 0x7ff0a00d0ad0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 --2- 172.21.15.116:0/669948472 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a00cc200 0x7ff0a00b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 -- 172.21.15.116:0/669948472 >> 172.21.15.116:0/669948472 conn(0x7ff0a006ea80 msgr2=0x7ff0a006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:32.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 -- 172.21.15.116:0/669948472 shutdown_connections 2023-11-08T08:21:32.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.899+0000 7ff0a603f700 1 -- 172.21.15.116:0/669948472 wait complete. 2023-11-08T08:21:32.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.900+0000 7ff0a603f700 1 Processor -- start 2023-11-08T08:21:32.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 -- start start 2023-11-08T08:21:32.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 0x7ff0a014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a00cc200 0x7ff0a00c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a015a5f0 0x7ff0a006bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00c8e10 con 0x7ff0a00b2d10 2023-11-08T08:21:32.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00c8850 con 0x7ff0a00cc200 2023-11-08T08:21:32.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff0a603f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00c8b30 con 0x7ff0a015a5f0 2023-11-08T08:21:32.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff09f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 0x7ff0a014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff09f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 0x7ff0a014a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47722/0 (socket says 172.21.15.116:47722) 2023-11-08T08:21:32.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.901+0000 7ff09f7fe700 1 -- 172.21.15.116:0/3090960367 learned_addr learned my addr 172.21.15.116:0/3090960367 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:32.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09effd700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a00cc200 0x7ff0a00c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09ffff700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a015a5f0 0x7ff0a006bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09f7fe700 1 -- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a015a5f0 msgr2=0x7ff0a006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09f7fe700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a015a5f0 0x7ff0a006bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09f7fe700 1 -- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a00cc200 msgr2=0x7ff0a00c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09f7fe700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a00cc200 0x7ff0a00c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09f7fe700 1 -- 172.21.15.116:0/3090960367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0940094a0 con 0x7ff0a00b2d10 2023-11-08T08:21:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09ffff700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a015a5f0 0x7ff0a006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff09f7fe700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 0x7ff0a014a880 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff094009fd0 tx=0x7ff094004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0940045d0 con 0x7ff0a00b2d10 2023-11-08T08:21:32.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09401f070 con 0x7ff0a00b2d10 2023-11-08T08:21:32.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a00ccf50 con 0x7ff0a00b2d10 2023-11-08T08:21:32.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0940038e0 con 0x7ff0a00b2d10 2023-11-08T08:21:32.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.902+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a00ccc70 con 0x7ff0a00b2d10 2023-11-08T08:21:32.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.903+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a00cd230 con 0x7ff0a00b2d10 2023-11-08T08:21:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.903+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff09400f050 con 0x7ff0a00b2d10 2023-11-08T08:21:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.904+0000 7ff087fff700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff08c0682f0 0x7ff08c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.904+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff094021070 con 0x7ff0a00b2d10 2023-11-08T08:21:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.904+0000 7ff09effd700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff08c0682f0 0x7ff08c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.904+0000 7ff09effd700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff08c0682f0 0x7ff08c06a7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00b6360 tx=0x7ff090009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:32.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:32.908+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff094053de0 con 0x7ff0a00b2d10 2023-11-08T08:21:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:33.133+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff0a00743f0 con 0x7ff0a00b2d10 2023-11-08T08:21:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.106+0000 7ff087fff700 1 -- 172.21.15.116:0/3090960367 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff094057400 con 0x7ff0a00b2d10 2023-11-08T08:21:34.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff08c0682f0 msgr2=0x7ff08c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:34.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff08c0682f0 0x7ff08c06a7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00b6360 tx=0x7ff090009210 comp rx=0 tx=0).stop 2023-11-08T08:21:34.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 msgr2=0x7ff0a014a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:34.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 0x7ff0a014a880 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff094009fd0 tx=0x7ff094004750 comp rx=0 tx=0).stop 2023-11-08T08:21:34.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 shutdown_connections 2023-11-08T08:21:34.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff0a015a5f0 0x7ff0a006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0a00b2d10 0x7ff0a014a880 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff08c0682f0 0x7ff08c06a7a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 --2- 172.21.15.116:0/3090960367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0a00cc200 0x7ff0a00c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 >> 172.21.15.116:0/3090960367 conn(0x7ff0a006ea80 msgr2=0x7ff0a00cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 shutdown_connections 2023-11-08T08:21:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.109+0000 7ff0a603f700 1 -- 172.21.15.116:0/3090960367 wait complete. 2023-11-08T08:21:34.124 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-08T08:21:34.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-08T08:21:34.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.253+0000 7f7911f9f700 1 Processor -- start 2023-11-08T08:21:34.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.253+0000 7f7911f9f700 1 -- start start 2023-11-08T08:21:34.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f7911f9f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c1599b0 0x7f790c15d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:34.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f7911f9f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f790c0af0a0 0x7f790c15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f7911f9f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c159ec0 con 0x7f790c0b1650 2023-11-08T08:21:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f7911f9f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c15d8a0 con 0x7f790c0af0a0 2023-11-08T08:21:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f7911f9f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c15db80 con 0x7f790c1599b0 2023-11-08T08:21:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f790bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f790c0af0a0 0x7f790c15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f790b7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f790c0b1650 0x7f790c0b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55356/0 (socket says 172.21.15.116:55356) 2023-11-08T08:21:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.254+0000 7f790b7fe700 1 -- 172.21.15.116:0/2310670158 learned_addr learned my addr 172.21.15.116:0/2310670158 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790bfff700 1 -- 172.21.15.116:0/2310670158 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c1599b0 msgr2=0x7f790c15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790bfff700 1 --2- 172.21.15.116:0/2310670158 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c1599b0 0x7f790c15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790a7fc700 1 -- 172.21.15.116:0/2310670158 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555189836 0 0) 0x7f790c159ec0 con 0x7f790c0b1650 2023-11-08T08:21:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790bfff700 1 -- 172.21.15.116:0/2310670158 >> v1:172.21.15.116:6789/0 conn(0x7f790c0b1650 legacy=0x7f790c0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790bfff700 1 -- 172.21.15.116:0/2310670158 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c15de60 con 0x7f790c0af0a0 2023-11-08T08:21:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790bfff700 1 --2- 172.21.15.116:0/2310670158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0af0a0 0x7f790c15a350 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7904009d90 tx=0x7f790400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18a5bb4deef88bf server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790a7fc700 1 -- 172.21.15.116:0/2310670158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790400e070 con 0x7f790c0af0a0 2023-11-08T08:21:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790a7fc700 1 -- 172.21.15.116:0/2310670158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790400bbd0 con 0x7f790c0af0a0 2023-11-08T08:21:34.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f790a7fc700 1 -- 172.21.15.116:0/2310670158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904012430 con 0x7f790c0af0a0 2023-11-08T08:21:34.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f7911f9f700 1 -- 172.21.15.116:0/2310670158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0af0a0 msgr2=0x7f790c15a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:34.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.255+0000 7f7911f9f700 1 --2- 172.21.15.116:0/2310670158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0af0a0 0x7f790c15a350 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7904009d90 tx=0x7f790400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:34.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.256+0000 7f7911f9f700 1 -- 172.21.15.116:0/2310670158 shutdown_connections 2023-11-08T08:21:34.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.256+0000 7f7911f9f700 1 --2- 172.21.15.116:0/2310670158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0af0a0 0x7f790c15a350 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.256+0000 7f7911f9f700 1 --2- 172.21.15.116:0/2310670158 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c1599b0 0x7f790c15d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.256+0000 7f7911f9f700 1 -- 172.21.15.116:0/2310670158 >> 172.21.15.116:0/2310670158 conn(0x7f790c06c010 msgr2=0x7f790c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.256+0000 7f7911f9f700 1 -- 172.21.15.116:0/2310670158 shutdown_connections 2023-11-08T08:21:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.256+0000 7f7911f9f700 1 -- 172.21.15.116:0/2310670158 wait complete. 2023-11-08T08:21:34.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.257+0000 7f7911f9f700 1 Processor -- start 2023-11-08T08:21:34.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.257+0000 7f7911f9f700 1 -- start start 2023-11-08T08:21:34.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c0af0a0 0x7f790c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:34.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 0x7f790c0a84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:34.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790c14aef0 0x7f790c1f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:34.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0b3670 con 0x7f790c0b1650 2023-11-08T08:21:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0b38c0 con 0x7f790c14aef0 2023-11-08T08:21:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0b3ba0 con 0x7f790c0af0a0 2023-11-08T08:21:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 0x7f790c0a84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 0x7f790c0a84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47742/0 (socket says 172.21.15.116:47742) 2023-11-08T08:21:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 -- 172.21.15.116:0/4102509374 learned_addr learned my addr 172.21.15.116:0/4102509374 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:34.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790b7fe700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c0af0a0 0x7f790c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:34.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 -- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c0af0a0 msgr2=0x7f790c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:34.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c0af0a0 0x7f790c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 -- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790c14aef0 msgr2=0x7f790c1f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:34.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790c14aef0 0x7f790c1f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:34.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 -- 172.21.15.116:0/4102509374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904009a70 con 0x7f790c0b1650 2023-11-08T08:21:34.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790b7fe700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c0af0a0 0x7f790c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:34.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f790affd700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 0x7f790c0a84e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00b500 tx=0x7f78fc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:34.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc011830 con 0x7f790c0b1650 2023-11-08T08:21:34.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc0119b0 con 0x7f790c0b1650 2023-11-08T08:21:34.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c15a890 con 0x7f790c0b1650 2023-11-08T08:21:34.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc011d60 con 0x7f790c0b1650 2023-11-08T08:21:34.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.258+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c15d590 con 0x7f790c0b1650 2023-11-08T08:21:34.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.259+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c15aa20 con 0x7f790c0b1650 2023-11-08T08:21:34.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.260+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f78fc021070 con 0x7f790c0b1650 2023-11-08T08:21:34.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.260+0000 7f78f3fff700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78f80682f0 0x7f78f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:34.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.260+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f78fc085c00 con 0x7f790c0b1650 2023-11-08T08:21:34.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.260+0000 7f790b7fe700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78f80682f0 0x7f78f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:34.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.260+0000 7f790b7fe700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78f80682f0 0x7f78f806a7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7900007fd0 tx=0x7f7900009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:34.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.264+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78fc08b020 con 0x7f790c0b1650 2023-11-08T08:21:34.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:34.489+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f790c0743f0 con 0x7f790c0b1650 2023-11-08T08:21:35.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.479+0000 7f78f3fff700 1 -- 172.21.15.116:0/4102509374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f78fc015020 con 0x7f790c0b1650 2023-11-08T08:21:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78f80682f0 msgr2=0x7f78f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78f80682f0 0x7f78f806a7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7900007fd0 tx=0x7f7900009040 comp rx=0 tx=0).stop 2023-11-08T08:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 msgr2=0x7f790c0a84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 0x7f790c0a84e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00b500 tx=0x7f78fc00b8c0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 shutdown_connections 2023-11-08T08:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f78f80682f0 0x7f78f806a7a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790c14aef0 0x7f790c1f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f790c0af0a0 0x7f790c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 --2- 172.21.15.116:0/4102509374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c0b1650 0x7f790c0a84e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 >> 172.21.15.116:0/4102509374 conn(0x7f790c06c010 msgr2=0x7f790c15cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 shutdown_connections 2023-11-08T08:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.482+0000 7f7911f9f700 1 -- 172.21.15.116:0/4102509374 wait complete. 2023-11-08T08:21:35.498 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-08T08:21:35.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:35.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.623+0000 7f0bc65d9700 1 Processor -- start 2023-11-08T08:21:35.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.623+0000 7f0bc65d9700 1 -- start start 2023-11-08T08:21:35.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bc65d9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0bc0071f70 0x7f0bc00cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bc65d9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0bc00b4a30 0x7f0bc00b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bc65d9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00cfb00 con 0x7f0bc00b4660 2023-11-08T08:21:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bc65d9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc015aa20 con 0x7f0bc0071f70 2023-11-08T08:21:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bc65d9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc00cf4d0 con 0x7f0bc00b4a30 2023-11-08T08:21:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0bc0071f70 0x7f0bc00cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0bc0071f70 0x7f0bc00cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47752/0 (socket says 172.21.15.116:47752) 2023-11-08T08:21:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 -- 172.21.15.116:0/2791000266 learned_addr learned my addr 172.21.15.116:0/2791000266 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bc4b76700 1 --2- 172.21.15.116:0/2791000266 >> v2:172.21.15.129:3300/0 conn(0x7f0bc00b4a30 0x7f0bc00b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 -- 172.21.15.116:0/2791000266 >> v2:172.21.15.129:3300/0 conn(0x7f0bc00b4a30 msgr2=0x7f0bc00b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 --2- 172.21.15.116:0/2791000266 >> v2:172.21.15.129:3300/0 conn(0x7f0bc00b4a30 0x7f0bc00b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 -- 172.21.15.116:0/2791000266 >> v1:172.21.15.129:6789/0 conn(0x7f0bc00b4660 legacy=0x7f0bc006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.624+0000 7f0bbf7fe700 1 -- 172.21.15.116:0/2791000266 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc0072550 con 0x7f0bc0071f70 2023-11-08T08:21:35.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bbf7fe700 1 --2- 172.21.15.116:0/2791000266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc00cff90 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0073cf0 tx=0x7f0bb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a9c238b7765adc server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:35.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bbeffd700 1 -- 172.21.15.116:0/2791000266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000b870 con 0x7f0bc0071f70 2023-11-08T08:21:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bbeffd700 1 -- 172.21.15.116:0/2791000266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000b9d0 con 0x7f0bc0071f70 2023-11-08T08:21:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bbeffd700 1 -- 172.21.15.116:0/2791000266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000bd60 con 0x7f0bc0071f70 2023-11-08T08:21:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2791000266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 msgr2=0x7f0bc00cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2791000266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc00cff90 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0073cf0 tx=0x7f0bb0009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2791000266 shutdown_connections 2023-11-08T08:21:35.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2791000266 >> v2:172.21.15.129:3300/0 conn(0x7f0bc00b4a30 0x7f0bc00b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2791000266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc00cff90 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2791000266 >> 172.21.15.116:0/2791000266 conn(0x7f0bc00b2fc0 msgr2=0x7f0bc0071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2791000266 shutdown_connections 2023-11-08T08:21:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.625+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2791000266 wait complete. 2023-11-08T08:21:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.626+0000 7f0bc65d9700 1 Processor -- start 2023-11-08T08:21:35.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 -- start start 2023-11-08T08:21:35.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0bc00b4660 0x7f0bc01f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0bc00b4a30 0x7f0bc01f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc01f66e0 con 0x7f0bc0071f70 2023-11-08T08:21:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc01f3170 con 0x7f0bc00b4660 2023-11-08T08:21:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bc65d9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00cd6f0 con 0x7f0bc00b4a30 2023-11-08T08:21:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bbffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bbffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc006da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47758/0 (socket says 172.21.15.116:47758) 2023-11-08T08:21:35.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.627+0000 7f0bbffff700 1 -- 172.21.15.116:0/2881796109 learned_addr learned my addr 172.21.15.116:0/2881796109 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:35.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbf7fe700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0bc00b4660 0x7f0bc01f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bc4b76700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0bc00b4a30 0x7f0bc01f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbffff700 1 -- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0bc00b4a30 msgr2=0x7f0bc01f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbffff700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0bc00b4a30 0x7f0bc01f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbffff700 1 -- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0bc00b4660 msgr2=0x7f0bc01f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbffff700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0bc00b4660 0x7f0bc01f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbffff700 1 -- 172.21.15.116:0/2881796109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb0009550 con 0x7f0bc0071f70 2023-11-08T08:21:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbffff700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc006da60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400c490 tx=0x7f0bb400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400cea0 con 0x7f0bc0071f70 2023-11-08T08:21:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb40052f0 con 0x7f0bc0071f70 2023-11-08T08:21:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc015f170 con 0x7f0bc0071f70 2023-11-08T08:21:35.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4006860 con 0x7f0bc0071f70 2023-11-08T08:21:35.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.628+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc00cf790 con 0x7f0bc0071f70 2023-11-08T08:21:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.629+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc015f300 con 0x7f0bc0071f70 2023-11-08T08:21:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.629+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0bb4010050 con 0x7f0bc0071f70 2023-11-08T08:21:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.630+0000 7f0bbcff9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bac0682f0 0x7f0bac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.630+0000 7f0bbf7fe700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bac0682f0 0x7f0bac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.630+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0bb4084cd0 con 0x7f0bc0071f70 2023-11-08T08:21:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.630+0000 7f0bbf7fe700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bac0682f0 0x7f0bac06a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0000c00 tx=0x7f0bb0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.634+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0bb404a5e0 con 0x7f0bc0071f70 2023-11-08T08:21:35.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.856+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bc00743f0 con 0x7f0bc0071f70 2023-11-08T08:21:35.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.857+0000 7f0bbcff9700 1 -- 172.21.15.116:0/2881796109 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4661 (secure 0 0 0) 0x7f0bb4051530 con 0x7f0bc0071f70 2023-11-08T08:21:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:35.859 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:35.475147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:35.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bac0682f0 msgr2=0x7f0bac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bac0682f0 0x7f0bac06a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0000c00 tx=0x7f0bb0008030 comp rx=0 tx=0).stop 2023-11-08T08:21:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 msgr2=0x7f0bc006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc006da60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400c490 tx=0x7f0bb400c7a0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 shutdown_connections 2023-11-08T08:21:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0bc00b4a30 0x7f0bc01f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc0071f70 0x7f0bc006da60 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f0bac0682f0 0x7f0bac06a7a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 --2- 172.21.15.116:0/2881796109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0bc00b4660 0x7f0bc01f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 >> 172.21.15.116:0/2881796109 conn(0x7f0bc00b2fc0 msgr2=0x7f0bc00c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 shutdown_connections 2023-11-08T08:21:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.859+0000 7f0bc65d9700 1 -- 172.21.15.116:0/2881796109 wait complete. 2023-11-08T08:21:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2023-11-08T08:21:35.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 Processor -- start 2023-11-08T08:21:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 -- start start 2023-11-08T08:21:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06a270 0x7fb8ac06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb8ac15cc80 0x7fb8ac06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8ac06e3a0 0x7fb8ac06ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac0b2490 con 0x7fb8ac06e3a0 2023-11-08T08:21:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac06ab80 con 0x7fb8ac15cc80 2023-11-08T08:21:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8b1c71700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0b1810 con 0x7fb8ac06a270 2023-11-08T08:21:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.992+0000 7fb8abfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8ac06e3a0 0x7fb8ac06ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8ac06e3a0 0x7fb8ac06ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47766/0 (socket says 172.21.15.116:47766) 2023-11-08T08:21:35.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 -- 172.21.15.116:0/3121665296 learned_addr learned my addr 172.21.15.116:0/3121665296 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:35.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8aaffd700 1 --2- 172.21.15.116:0/3121665296 >> v2:172.21.15.129:3300/0 conn(0x7fb8ac15cc80 0x7fb8ac06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8ab7fe700 1 --2- 172.21.15.116:0/3121665296 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06a270 0x7fb8ac06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:35.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 -- 172.21.15.116:0/3121665296 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06a270 msgr2=0x7fb8ac06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:35.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 --2- 172.21.15.116:0/3121665296 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06a270 0x7fb8ac06a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 -- 172.21.15.116:0/3121665296 >> v2:172.21.15.129:3300/0 conn(0x7fb8ac15cc80 msgr2=0x7fb8ac06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 --2- 172.21.15.116:0/3121665296 >> v2:172.21.15.129:3300/0 conn(0x7fb8ac15cc80 0x7fb8ac06de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 -- 172.21.15.116:0/3121665296 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0b1af0 con 0x7fb8ac06e3a0 2023-11-08T08:21:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8abfff700 1 --2- 172.21.15.116:0/3121665296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac06e3a0 0x7fb8ac06ce20 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0d0df0 tx=0x7fb8a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5e9fe18b2bfefa server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8aa7fc700 1 -- 172.21.15.116:0/3121665296 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400e070 con 0x7fb8ac06e3a0 2023-11-08T08:21:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8aa7fc700 1 -- 172.21.15.116:0/3121665296 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a400b9c0 con 0x7fb8ac06e3a0 2023-11-08T08:21:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.993+0000 7fb8aa7fc700 1 -- 172.21.15.116:0/3121665296 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400bd70 con 0x7fb8ac06e3a0 2023-11-08T08:21:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 -- 172.21.15.116:0/3121665296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac06e3a0 msgr2=0x7fb8ac06ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/3121665296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac06e3a0 0x7fb8ac06ce20 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0d0df0 tx=0x7fb8a400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 -- 172.21.15.116:0/3121665296 shutdown_connections 2023-11-08T08:21:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/3121665296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac06e3a0 0x7fb8ac06ce20 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/3121665296 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06a270 0x7fb8ac06a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/3121665296 >> v2:172.21.15.129:3300/0 conn(0x7fb8ac15cc80 0x7fb8ac06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 -- 172.21.15.116:0/3121665296 >> 172.21.15.116:0/3121665296 conn(0x7fb8ac06c010 msgr2=0x7fb8ac0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 -- 172.21.15.116:0/3121665296 shutdown_connections 2023-11-08T08:21:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.994+0000 7fb8b1c71700 1 -- 172.21.15.116:0/3121665296 wait complete. 2023-11-08T08:21:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.995+0000 7fb8b1c71700 1 Processor -- start 2023-11-08T08:21:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.995+0000 7fb8b1c71700 1 -- start start 2023-11-08T08:21:36.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8ac06a270 0x7fb8ac0c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06e3a0 0x7fb8ac0c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 0x7fb8ac0c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac1f2ee0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac1f2360 con 0x7fb8ac06a270 2023-11-08T08:21:36.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac1f2640 con 0x7fb8ac06e3a0 2023-11-08T08:21:36.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 0x7fb8ac0c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 0x7fb8ac0c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47780/0 (socket says 172.21.15.116:47780) 2023-11-08T08:21:36.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 -- 172.21.15.116:0/1030569917 learned_addr learned my addr 172.21.15.116:0/1030569917 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:36.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8ab7fe700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8ac06a270 0x7fb8ac0c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8aaffd700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06e3a0 0x7fb8ac0c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 -- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06e3a0 msgr2=0x7fb8ac0c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06e3a0 0x7fb8ac0c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 -- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8ac06a270 msgr2=0x7fb8ac0c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8ac06a270 0x7fb8ac0c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 -- 172.21.15.116:0/1030569917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a4009b30 con 0x7fb8ac15cc80 2023-11-08T08:21:36.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8abfff700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 0x7fb8ac0c8690 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac067450 tx=0x7fb8a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a40092e0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4004250 con 0x7fb8ac15cc80 2023-11-08T08:21:36.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac0b20b0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4004620 con 0x7fb8ac15cc80 2023-11-08T08:21:36.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.996+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac0b1dd0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.997+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac06e7e0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.998+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb8a402b070 con 0x7fb8ac15cc80 2023-11-08T08:21:36.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.998+0000 7fb8a3fff700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb88c0682f0 0x7fb88c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.998+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb8a4086cc0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.998+0000 7fb8ab7fe700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb88c0682f0 0x7fb88c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:35.999+0000 7fb8ab7fe700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb88c0682f0 0x7fb88c06a7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0c3550 tx=0x7fb894008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.002+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb8a40534a0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.181+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8ac0743f0 con 0x7fb8ac15cc80 2023-11-08T08:21:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.182+0000 7fb8a3fff700 1 -- 172.21.15.116:0/1030569917 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17959 (secure 0 0 0) 0x7fb8a401f090 con 0x7fb8ac15cc80 2023-11-08T08:21:36.184 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:36.184 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:33.084884+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb88c0682f0 msgr2=0x7fb88c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb88c0682f0 0x7fb88c06a7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0c3550 tx=0x7fb894008040 comp rx=0 tx=0).stop 2023-11-08T08:21:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 msgr2=0x7fb8ac0c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 0x7fb8ac0c8690 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac067450 tx=0x7fb8a4004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 shutdown_connections 2023-11-08T08:21:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8ac15cc80 0x7fb8ac0c8690 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb88c0682f0 0x7fb88c06a7a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8ac06a270 0x7fb8ac0c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 --2- 172.21.15.116:0/1030569917 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb8ac06e3a0 0x7fb8ac0c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 >> 172.21.15.116:0/1030569917 conn(0x7fb8ac06c010 msgr2=0x7fb8ac0ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 shutdown_connections 2023-11-08T08:21:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.185+0000 7fb8b1c71700 1 -- 172.21.15.116:0/1030569917 wait complete. 2023-11-08T08:21:36.198 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-08T08:21:36.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:21:36.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.317+0000 7ff63b8b9700 1 Processor -- start 2023-11-08T08:21:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.317+0000 7ff63b8b9700 1 -- start start 2023-11-08T08:21:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff63b8b9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff63406b7b0 0x7ff634072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff63b8b9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6340ad1f0 0x7ff6340ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff63b8b9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6340726c0 con 0x7ff6340b0880 2023-11-08T08:21:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff63b8b9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63406bc50 con 0x7ff6340ad1f0 2023-11-08T08:21:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff63b8b9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6340adb50 con 0x7ff63406b7b0 2023-11-08T08:21:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6340ad1f0 0x7ff6340ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6340ad1f0 0x7ff6340ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47790/0 (socket says 172.21.15.116:47790) 2023-11-08T08:21:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 -- 172.21.15.116:0/3248377820 learned_addr learned my addr 172.21.15.116:0/3248377820 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff638e54700 1 --2- 172.21.15.116:0/3248377820 >> v2:172.21.15.129:3300/0 conn(0x7ff63406b7b0 0x7ff634072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 -- 172.21.15.116:0/3248377820 >> v2:172.21.15.129:3300/0 conn(0x7ff63406b7b0 msgr2=0x7ff634072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 --2- 172.21.15.116:0/3248377820 >> v2:172.21.15.129:3300/0 conn(0x7ff63406b7b0 0x7ff634072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 -- 172.21.15.116:0/3248377820 >> v1:172.21.15.129:6789/0 conn(0x7ff6340b0880 legacy=0x7ff6340aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:36.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff639e56700 1 -- 172.21.15.116:0/3248377820 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6340b43f0 con 0x7ff6340ad1f0 2023-11-08T08:21:36.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.318+0000 7ff638e54700 1 --2- 172.21.15.116:0/3248377820 >> v2:172.21.15.129:3300/0 conn(0x7ff63406b7b0 0x7ff634072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:36.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff639e56700 1 --2- 172.21.15.116:0/3248377820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6340ad1f0 0x7ff6340ad610 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff63406f560 tx=0x7ff62c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7b2b934eea9425 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff62bfff700 1 -- 172.21.15.116:0/3248377820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c00e070 con 0x7ff6340ad1f0 2023-11-08T08:21:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff62bfff700 1 -- 172.21.15.116:0/3248377820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c003a20 con 0x7ff6340ad1f0 2023-11-08T08:21:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff62bfff700 1 -- 172.21.15.116:0/3248377820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c003dd0 con 0x7ff6340ad1f0 2023-11-08T08:21:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 -- 172.21.15.116:0/3248377820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6340ad1f0 msgr2=0x7ff6340ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/3248377820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6340ad1f0 0x7ff6340ad610 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff63406f560 tx=0x7ff62c00b330 comp rx=0 tx=0).stop 2023-11-08T08:21:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 -- 172.21.15.116:0/3248377820 shutdown_connections 2023-11-08T08:21:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/3248377820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6340ad1f0 0x7ff6340ad610 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/3248377820 >> v2:172.21.15.129:3300/0 conn(0x7ff63406b7b0 0x7ff634072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 -- 172.21.15.116:0/3248377820 >> 172.21.15.116:0/3248377820 conn(0x7ff6340b2fc0 msgr2=0x7ff634071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:36.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 -- 172.21.15.116:0/3248377820 shutdown_connections 2023-11-08T08:21:36.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.319+0000 7ff63b8b9700 1 -- 172.21.15.116:0/3248377820 wait complete. 2023-11-08T08:21:36.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.320+0000 7ff63b8b9700 1 Processor -- start 2023-11-08T08:21:36.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 -- start start 2023-11-08T08:21:36.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 0x7ff63415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6340ad1f0 0x7ff6340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6340b0880 0x7ff6341f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6341f2030 con 0x7ff63406b7b0 2023-11-08T08:21:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63409b490 con 0x7ff6340b0880 2023-11-08T08:21:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff63b8b9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63409b770 con 0x7ff6340ad1f0 2023-11-08T08:21:36.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff639655700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 0x7ff63415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff639655700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 0x7ff63415c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47804/0 (socket says 172.21.15.116:47804) 2023-11-08T08:21:36.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff639655700 1 -- 172.21.15.116:0/6576451 learned_addr learned my addr 172.21.15.116:0/6576451 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:36.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff639e56700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6340b0880 0x7ff6341f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.321+0000 7ff638e54700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6340ad1f0 0x7ff6340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff639655700 1 -- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6340ad1f0 msgr2=0x7ff6340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff639655700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6340ad1f0 0x7ff6340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff639655700 1 -- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6340b0880 msgr2=0x7ff6341f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff639655700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6340b0880 0x7ff6341f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff639655700 1 -- 172.21.15.116:0/6576451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c009b30 con 0x7ff63406b7b0 2023-11-08T08:21:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff639655700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 0x7ff63415c4d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c4d0 tx=0x7ff62400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624011790 con 0x7ff63406b7b0 2023-11-08T08:21:36.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6240118f0 con 0x7ff63406b7b0 2023-11-08T08:21:36.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6340b4d00 con 0x7ff63406b7b0 2023-11-08T08:21:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624011ca0 con 0x7ff63406b7b0 2023-11-08T08:21:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.322+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6340b46d0 con 0x7ff63406b7b0 2023-11-08T08:21:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.323+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6340b4fe0 con 0x7ff63406b7b0 2023-11-08T08:21:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.324+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff624010050 con 0x7ff63406b7b0 2023-11-08T08:21:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.324+0000 7ff629ffb700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6180682f0 0x7ff61806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.324+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff6240858e0 con 0x7ff63406b7b0 2023-11-08T08:21:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.324+0000 7ff638e54700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6180682f0 0x7ff61806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.324+0000 7ff638e54700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6180682f0 0x7ff61806a7a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.328+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff624052140 con 0x7ff63406b7b0 2023-11-08T08:21:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.509+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6341f2170 con 0x7ff6180682f0 2023-11-08T08:21:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.517+0000 7ff629ffb700 1 -- 172.21.15.116:0/6576451 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311650 (secure 0 0 0) 0x7ff63000a720 con 0x7ff6180682f0 2023-11-08T08:21:36.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6180682f0 msgr2=0x7ff61806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6180682f0 0x7ff61806a7a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630008040 comp rx=0 tx=0).stop 2023-11-08T08:21:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 msgr2=0x7ff63415c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 0x7ff63415c4d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c4d0 tx=0x7ff62400c7e0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 shutdown_connections 2023-11-08T08:21:36.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7ff6180682f0 0x7ff61806a7a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6340b0880 0x7ff6341f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63406b7b0 0x7ff63415c4d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 --2- 172.21.15.116:0/6576451 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6340ad1f0 0x7ff6340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 >> 172.21.15.116:0/6576451 conn(0x7ff6340b2fc0 msgr2=0x7ff63406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 shutdown_connections 2023-11-08T08:21:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.521+0000 7ff63b8b9700 1 -- 172.21.15.116:0/6576451 wait complete. 2023-11-08T08:21:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:21:36.534 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2023-11-08T08:21:36.391323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":77636,"kb_used_data":6468,"kb_used_omap":0,"kb_used_meta":71104,"kb_avail":754897084,"statfs":{"total":773094113280,"available":773014614016,"internally_reserved":0,"allocated":6623232,"data_stored":3525647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72810496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.709177"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099377+0000","last_change":"2023-11-08T08:21:32.107547+0000","last_active":"2023-11-08T08:21:33.099377+0000","last_peered":"2023-11-08T08:21:33.099377+0000","last_clean":"2023-11-08T08:21:33.099377+0000","last_became_active":"2023-11-08T08:21:32.107232+0000","last_became_peered":"2023-11-08T08:21:32.107232+0000","last_unstale":"2023-11-08T08:21:33.099377+0000","last_undegraded":"2023-11-08T08:21:33.099377+0000","last_fullsized":"2023-11-08T08:21:33.099377+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:51.265070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291215+0000","last_change":"2023-11-08T08:21:32.100334+0000","last_active":"2023-11-08T08:21:33.291215+0000","last_peered":"2023-11-08T08:21:33.291215+0000","last_clean":"2023-11-08T08:21:33.291215+0000","last_became_active":"2023-11-08T08:21:32.100061+0000","last_became_peered":"2023-11-08T08:21:32.100061+0000","last_unstale":"2023-11-08T08:21:33.291215+0000","last_undegraded":"2023-11-08T08:21:33.291215+0000","last_fullsized":"2023-11-08T08:21:33.291215+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:40:41.657914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290752+0000","last_change":"2023-11-08T08:21:32.103981+0000","last_active":"2023-11-08T08:21:33.290752+0000","last_peered":"2023-11-08T08:21:33.290752+0000","last_clean":"2023-11-08T08:21:33.290752+0000","last_became_active":"2023-11-08T08:21:32.103757+0000","last_became_peered":"2023-11-08T08:21:32.103757+0000","last_unstale":"2023-11-08T08:21:33.290752+0000","last_undegraded":"2023-11-08T08:21:33.290752+0000","last_fullsized":"2023-11-08T08:21:33.290752+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:57:30.530999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291052+0000","last_change":"2023-11-08T08:21:32.106439+0000","last_active":"2023-11-08T08:21:33.291052+0000","last_peered":"2023-11-08T08:21:33.291052+0000","last_clean":"2023-11-08T08:21:33.291052+0000","last_became_active":"2023-11-08T08:21:32.103271+0000","last_became_peered":"2023-11-08T08:21:32.103271+0000","last_unstale":"2023-11-08T08:21:33.291052+0000","last_undegraded":"2023-11-08T08:21:33.291052+0000","last_fullsized":"2023-11-08T08:21:33.291052+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:43:31.204987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290696+0000","last_change":"2023-11-08T08:21:32.102308+0000","last_active":"2023-11-08T08:21:33.290696+0000","last_peered":"2023-11-08T08:21:33.290696+0000","last_clean":"2023-11-08T08:21:33.290696+0000","last_became_active":"2023-11-08T08:21:32.102029+0000","last_became_peered":"2023-11-08T08:21:32.102029+0000","last_unstale":"2023-11-08T08:21:33.290696+0000","last_undegraded":"2023-11-08T08:21:33.290696+0000","last_fullsized":"2023-11-08T08:21:33.290696+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:44.888334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290708+0000","last_change":"2023-11-08T08:21:32.107208+0000","last_active":"2023-11-08T08:21:33.290708+0000","last_peered":"2023-11-08T08:21:33.290708+0000","last_clean":"2023-11-08T08:21:33.290708+0000","last_became_active":"2023-11-08T08:21:32.106514+0000","last_became_peered":"2023-11-08T08:21:32.106514+0000","last_unstale":"2023-11-08T08:21:33.290708+0000","last_undegraded":"2023-11-08T08:21:33.290708+0000","last_fullsized":"2023-11-08T08:21:33.290708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:26.497082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098938+0000","last_change":"2023-11-08T08:21:32.111545+0000","last_active":"2023-11-08T08:21:33.098938+0000","last_peered":"2023-11-08T08:21:33.098938+0000","last_clean":"2023-11-08T08:21:33.098938+0000","last_became_active":"2023-11-08T08:21:32.111179+0000","last_became_peered":"2023-11-08T08:21:32.111179+0000","last_unstale":"2023-11-08T08:21:33.098938+0000","last_undegraded":"2023-11-08T08:21:33.098938+0000","last_fullsized":"2023-11-08T08:21:33.098938+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:59.550762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290266+0000","last_change":"2023-11-08T08:21:32.099428+0000","last_active":"2023-11-08T08:21:33.290266+0000","last_peered":"2023-11-08T08:21:33.290266+0000","last_clean":"2023-11-08T08:21:33.290266+0000","last_became_active":"2023-11-08T08:21:32.098595+0000","last_became_peered":"2023-11-08T08:21:32.098595+0000","last_unstale":"2023-11-08T08:21:33.290266+0000","last_undegraded":"2023-11-08T08:21:33.290266+0000","last_fullsized":"2023-11-08T08:21:33.290266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:31.579603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291320+0000","last_change":"2023-11-08T08:21:32.108391+0000","last_active":"2023-11-08T08:21:33.291320+0000","last_peered":"2023-11-08T08:21:33.291320+0000","last_clean":"2023-11-08T08:21:33.291320+0000","last_became_active":"2023-11-08T08:21:32.107821+0000","last_became_peered":"2023-11-08T08:21:32.107821+0000","last_unstale":"2023-11-08T08:21:33.291320+0000","last_undegraded":"2023-11-08T08:21:33.291320+0000","last_fullsized":"2023-11-08T08:21:33.291320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:58:56.180020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098609+0000","last_change":"2023-11-08T08:21:32.105640+0000","last_active":"2023-11-08T08:21:33.098609+0000","last_peered":"2023-11-08T08:21:33.098609+0000","last_clean":"2023-11-08T08:21:33.098609+0000","last_became_active":"2023-11-08T08:21:32.105386+0000","last_became_peered":"2023-11-08T08:21:32.105386+0000","last_unstale":"2023-11-08T08:21:33.098609+0000","last_undegraded":"2023-11-08T08:21:33.098609+0000","last_fullsized":"2023-11-08T08:21:33.098609+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:39:47.423926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291075+0000","last_change":"2023-11-08T08:21:32.110541+0000","last_active":"2023-11-08T08:21:33.291075+0000","last_peered":"2023-11-08T08:21:33.291075+0000","last_clean":"2023-11-08T08:21:33.291075+0000","last_became_active":"2023-11-08T08:21:32.109913+0000","last_became_peered":"2023-11-08T08:21:32.109913+0000","last_unstale":"2023-11-08T08:21:33.291075+0000","last_undegraded":"2023-11-08T08:21:33.291075+0000","last_fullsized":"2023-11-08T08:21:33.291075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:44:47.641703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290963+0000","last_change":"2023-11-08T08:21:32.110027+0000","last_active":"2023-11-08T08:21:33.290963+0000","last_peered":"2023-11-08T08:21:33.290963+0000","last_clean":"2023-11-08T08:21:33.290963+0000","last_became_active":"2023-11-08T08:21:32.109655+0000","last_became_peered":"2023-11-08T08:21:32.109655+0000","last_unstale":"2023-11-08T08:21:33.290963+0000","last_undegraded":"2023-11-08T08:21:33.290963+0000","last_fullsized":"2023-11-08T08:21:33.290963+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:49:49.217341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096364+0000","last_change":"2023-11-08T08:21:32.101094+0000","last_active":"2023-11-08T08:21:33.096364+0000","last_peered":"2023-11-08T08:21:33.096364+0000","last_clean":"2023-11-08T08:21:33.096364+0000","last_became_active":"2023-11-08T08:21:32.100844+0000","last_became_peered":"2023-11-08T08:21:32.100844+0000","last_unstale":"2023-11-08T08:21:33.096364+0000","last_undegraded":"2023-11-08T08:21:33.096364+0000","last_fullsized":"2023-11-08T08:21:33.096364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:00:03.701169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096614+0000","last_change":"2023-11-08T08:21:32.100673+0000","last_active":"2023-11-08T08:21:33.096614+0000","last_peered":"2023-11-08T08:21:33.096614+0000","last_clean":"2023-11-08T08:21:33.096614+0000","last_became_active":"2023-11-08T08:21:32.100410+0000","last_became_peered":"2023-11-08T08:21:32.100410+0000","last_unstale":"2023-11-08T08:21:33.096614+0000","last_undegraded":"2023-11-08T08:21:33.096614+0000","last_fullsized":"2023-11-08T08:21:33.096614+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:16.672240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098327+0000","last_change":"2023-11-08T08:21:32.111428+0000","last_active":"2023-11-08T08:21:33.098327+0000","last_peered":"2023-11-08T08:21:33.098327+0000","last_clean":"2023-11-08T08:21:33.098327+0000","last_became_active":"2023-11-08T08:21:32.111166+0000","last_became_peered":"2023-11-08T08:21:32.111166+0000","last_unstale":"2023-11-08T08:21:33.098327+0000","last_undegraded":"2023-11-08T08:21:33.098327+0000","last_fullsized":"2023-11-08T08:21:33.098327+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:43:31.204987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290838+0000","last_change":"2023-11-08T08:21:32.097392+0000","last_active":"2023-11-08T08:21:33.290838+0000","last_peered":"2023-11-08T08:21:33.290838+0000","last_clean":"2023-11-08T08:21:33.290838+0000","last_became_active":"2023-11-08T08:21:32.097119+0000","last_became_peered":"2023-11-08T08:21:32.097119+0000","last_unstale":"2023-11-08T08:21:33.290838+0000","last_undegraded":"2023-11-08T08:21:33.290838+0000","last_fullsized":"2023-11-08T08:21:33.290838+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:16.672240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176453+0000","last_change":"2023-11-08T08:21:32.106868+0000","last_active":"2023-11-08T08:21:33.176453+0000","last_peered":"2023-11-08T08:21:33.176453+0000","last_clean":"2023-11-08T08:21:33.176453+0000","last_became_active":"2023-11-08T08:21:32.106601+0000","last_became_peered":"2023-11-08T08:21:32.106601+0000","last_unstale":"2023-11-08T08:21:33.176453+0000","last_undegraded":"2023-11-08T08:21:33.176453+0000","last_fullsized":"2023-11-08T08:21:33.176453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:21.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291349+0000","last_change":"2023-11-08T08:21:32.109424+0000","last_active":"2023-11-08T08:21:33.291349+0000","last_peered":"2023-11-08T08:21:33.291349+0000","last_clean":"2023-11-08T08:21:33.291349+0000","last_became_active":"2023-11-08T08:21:32.109178+0000","last_became_peered":"2023-11-08T08:21:32.109178+0000","last_unstale":"2023-11-08T08:21:33.291349+0000","last_undegraded":"2023-11-08T08:21:33.291349+0000","last_fullsized":"2023-11-08T08:21:33.291349+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:44:15.097568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291171+0000","last_change":"2023-11-08T08:21:32.110653+0000","last_active":"2023-11-08T08:21:33.291171+0000","last_peered":"2023-11-08T08:21:33.291171+0000","last_clean":"2023-11-08T08:21:33.291171+0000","last_became_active":"2023-11-08T08:21:32.110387+0000","last_became_peered":"2023-11-08T08:21:32.110387+0000","last_unstale":"2023-11-08T08:21:33.291171+0000","last_undegraded":"2023-11-08T08:21:33.291171+0000","last_fullsized":"2023-11-08T08:21:33.291171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:47:17.820009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176288+0000","last_change":"2023-11-08T08:21:32.104375+0000","last_active":"2023-11-08T08:21:33.176288+0000","last_peered":"2023-11-08T08:21:33.176288+0000","last_clean":"2023-11-08T08:21:33.176288+0000","last_became_active":"2023-11-08T08:21:32.104132+0000","last_became_peered":"2023-11-08T08:21:32.104132+0000","last_unstale":"2023-11-08T08:21:33.176288+0000","last_undegraded":"2023-11-08T08:21:33.176288+0000","last_fullsized":"2023-11-08T08:21:33.176288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:44.888334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096390+0000","last_change":"2023-11-08T08:21:32.099032+0000","last_active":"2023-11-08T08:21:33.096390+0000","last_peered":"2023-11-08T08:21:33.096390+0000","last_clean":"2023-11-08T08:21:33.096390+0000","last_became_active":"2023-11-08T08:21:32.098175+0000","last_became_peered":"2023-11-08T08:21:32.098175+0000","last_unstale":"2023-11-08T08:21:33.096390+0000","last_undegraded":"2023-11-08T08:21:33.096390+0000","last_fullsized":"2023-11-08T08:21:33.096390+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:28:33.365580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290585+0000","last_change":"2023-11-08T08:21:32.107442+0000","last_active":"2023-11-08T08:21:33.290585+0000","last_peered":"2023-11-08T08:21:33.290585+0000","last_clean":"2023-11-08T08:21:33.290585+0000","last_became_active":"2023-11-08T08:21:32.106950+0000","last_became_peered":"2023-11-08T08:21:32.106950+0000","last_unstale":"2023-11-08T08:21:33.290585+0000","last_undegraded":"2023-11-08T08:21:33.290585+0000","last_fullsized":"2023-11-08T08:21:33.290585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:58:56.180020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290913+0000","last_change":"2023-11-08T08:21:32.102073+0000","last_active":"2023-11-08T08:21:33.290913+0000","last_peered":"2023-11-08T08:21:33.290913+0000","last_clean":"2023-11-08T08:21:33.290913+0000","last_became_active":"2023-11-08T08:21:32.101799+0000","last_became_peered":"2023-11-08T08:21:32.101799+0000","last_unstale":"2023-11-08T08:21:33.290913+0000","last_undegraded":"2023-11-08T08:21:33.290913+0000","last_fullsized":"2023-11-08T08:21:33.290913+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:51.912547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290340+0000","last_change":"2023-11-08T08:21:32.100654+0000","last_active":"2023-11-08T08:21:33.290340+0000","last_peered":"2023-11-08T08:21:33.290340+0000","last_clean":"2023-11-08T08:21:33.290340+0000","last_became_active":"2023-11-08T08:21:32.100408+0000","last_became_peered":"2023-11-08T08:21:32.100408+0000","last_unstale":"2023-11-08T08:21:33.290340+0000","last_undegraded":"2023-11-08T08:21:33.290340+0000","last_fullsized":"2023-11-08T08:21:33.290340+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:32:16.748207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096568+0000","last_change":"2023-11-08T08:21:32.103128+0000","last_active":"2023-11-08T08:21:33.096568+0000","last_peered":"2023-11-08T08:21:33.096568+0000","last_clean":"2023-11-08T08:21:33.096568+0000","last_became_active":"2023-11-08T08:21:32.102847+0000","last_became_peered":"2023-11-08T08:21:32.102847+0000","last_unstale":"2023-11-08T08:21:33.096568+0000","last_undegraded":"2023-11-08T08:21:33.096568+0000","last_fullsized":"2023-11-08T08:21:33.096568+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:11:14.299828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290509+0000","last_change":"2023-11-08T08:21:32.104907+0000","last_active":"2023-11-08T08:21:33.290509+0000","last_peered":"2023-11-08T08:21:33.290509+0000","last_clean":"2023-11-08T08:21:33.290509+0000","last_became_active":"2023-11-08T08:21:32.104654+0000","last_became_peered":"2023-11-08T08:21:32.104654+0000","last_unstale":"2023-11-08T08:21:33.290509+0000","last_undegraded":"2023-11-08T08:21:33.290509+0000","last_fullsized":"2023-11-08T08:21:33.290509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:26:25.345947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096644+0000","last_change":"2023-11-08T08:21:32.103196+0000","last_active":"2023-11-08T08:21:33.096644+0000","last_peered":"2023-11-08T08:21:33.096644+0000","last_clean":"2023-11-08T08:21:33.096644+0000","last_became_active":"2023-11-08T08:21:32.102942+0000","last_became_peered":"2023-11-08T08:21:32.102942+0000","last_unstale":"2023-11-08T08:21:33.096644+0000","last_undegraded":"2023-11-08T08:21:33.096644+0000","last_fullsized":"2023-11-08T08:21:33.096644+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:51:10.113988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099380+0000","last_change":"2023-11-08T08:21:32.109134+0000","last_active":"2023-11-08T08:21:33.099380+0000","last_peered":"2023-11-08T08:21:33.099380+0000","last_clean":"2023-11-08T08:21:33.099380+0000","last_became_active":"2023-11-08T08:21:32.108810+0000","last_became_peered":"2023-11-08T08:21:32.108810+0000","last_unstale":"2023-11-08T08:21:33.099380+0000","last_undegraded":"2023-11-08T08:21:33.099380+0000","last_fullsized":"2023-11-08T08:21:33.099380+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:25:17.999948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291177+0000","last_change":"2023-11-08T08:21:32.107682+0000","last_active":"2023-11-08T08:21:33.291177+0000","last_peered":"2023-11-08T08:21:33.291177+0000","last_clean":"2023-11-08T08:21:33.291177+0000","last_became_active":"2023-11-08T08:21:32.107419+0000","last_became_peered":"2023-11-08T08:21:32.107419+0000","last_unstale":"2023-11-08T08:21:33.291177+0000","last_undegraded":"2023-11-08T08:21:33.291177+0000","last_fullsized":"2023-11-08T08:21:33.291177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:44:44.990866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291185+0000","last_change":"2023-11-08T08:21:32.107532+0000","last_active":"2023-11-08T08:21:33.291185+0000","last_peered":"2023-11-08T08:21:33.291185+0000","last_clean":"2023-11-08T08:21:33.291185+0000","last_became_active":"2023-11-08T08:21:32.106993+0000","last_became_peered":"2023-11-08T08:21:32.106993+0000","last_unstale":"2023-11-08T08:21:33.291185+0000","last_undegraded":"2023-11-08T08:21:33.291185+0000","last_fullsized":"2023-11-08T08:21:33.291185+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:40:41.657914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290580+0000","last_change":"2023-11-08T08:21:32.101225+0000","last_active":"2023-11-08T08:21:33.290580+0000","last_peered":"2023-11-08T08:21:33.290580+0000","last_clean":"2023-11-08T08:21:33.290580+0000","last_became_active":"2023-11-08T08:21:32.100444+0000","last_became_peered":"2023-11-08T08:21:32.100444+0000","last_unstale":"2023-11-08T08:21:33.290580+0000","last_undegraded":"2023-11-08T08:21:33.290580+0000","last_fullsized":"2023-11-08T08:21:33.290580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:15:59.123082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099029+0000","last_change":"2023-11-08T08:21:32.108546+0000","last_active":"2023-11-08T08:21:33.099029+0000","last_peered":"2023-11-08T08:21:33.099029+0000","last_clean":"2023-11-08T08:21:33.099029+0000","last_became_active":"2023-11-08T08:21:32.108262+0000","last_became_peered":"2023-11-08T08:21:32.108262+0000","last_unstale":"2023-11-08T08:21:33.099029+0000","last_undegraded":"2023-11-08T08:21:33.099029+0000","last_fullsized":"2023-11-08T08:21:33.099029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:32.238411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291040+0000","last_change":"2023-11-08T08:21:32.109010+0000","last_active":"2023-11-08T08:21:33.291040+0000","last_peered":"2023-11-08T08:21:33.291040+0000","last_clean":"2023-11-08T08:21:33.291040+0000","last_became_active":"2023-11-08T08:21:32.107037+0000","last_became_peered":"2023-11-08T08:21:32.107037+0000","last_unstale":"2023-11-08T08:21:33.291040+0000","last_undegraded":"2023-11-08T08:21:33.291040+0000","last_fullsized":"2023-11-08T08:21:33.291040+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:53:35.233536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1f","version":"55'4","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099434+0000","last_change":"2023-11-08T08:21:31.704889+0000","last_active":"2023-11-08T08:21:33.099434+0000","last_peered":"2023-11-08T08:21:33.099434+0000","last_clean":"2023-11-08T08:21:33.099434+0000","last_became_active":"2023-11-08T08:21:31.704617+0000","last_became_peered":"2023-11-08T08:21:31.704617+0000","last_unstale":"2023-11-08T08:21:33.099434+0000","last_undegraded":"2023-11-08T08:21:33.099434+0000","last_fullsized":"2023-11-08T08:21:33.099434+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:50.904536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290462+0000","last_change":"2023-11-08T08:21:32.102306+0000","last_active":"2023-11-08T08:21:33.290462+0000","last_peered":"2023-11-08T08:21:33.290462+0000","last_clean":"2023-11-08T08:21:33.290462+0000","last_became_active":"2023-11-08T08:21:32.102048+0000","last_became_peered":"2023-11-08T08:21:32.102048+0000","last_unstale":"2023-11-08T08:21:33.290462+0000","last_undegraded":"2023-11-08T08:21:33.290462+0000","last_fullsized":"2023-11-08T08:21:33.290462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:07.315328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1e","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099297+0000","last_change":"2023-11-08T08:21:31.706003+0000","last_active":"2023-11-08T08:21:33.099297+0000","last_peered":"2023-11-08T08:21:33.099297+0000","last_clean":"2023-11-08T08:21:33.099297+0000","last_became_active":"2023-11-08T08:21:31.705747+0000","last_became_peered":"2023-11-08T08:21:31.705747+0000","last_unstale":"2023-11-08T08:21:33.099297+0000","last_undegraded":"2023-11-08T08:21:33.099297+0000","last_fullsized":"2023-11-08T08:21:33.099297+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:25:14.455165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291244+0000","last_change":"2023-11-08T08:21:32.100306+0000","last_active":"2023-11-08T08:21:33.291244+0000","last_peered":"2023-11-08T08:21:33.291244+0000","last_clean":"2023-11-08T08:21:33.291244+0000","last_became_active":"2023-11-08T08:21:32.099999+0000","last_became_peered":"2023-11-08T08:21:32.099999+0000","last_unstale":"2023-11-08T08:21:33.291244+0000","last_undegraded":"2023-11-08T08:21:33.291244+0000","last_fullsized":"2023-11-08T08:21:33.291244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:44:10.321033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099583+0000","last_change":"2023-11-08T08:21:31.703084+0000","last_active":"2023-11-08T08:21:33.099583+0000","last_peered":"2023-11-08T08:21:33.099583+0000","last_clean":"2023-11-08T08:21:33.099583+0000","last_became_active":"2023-11-08T08:21:31.702806+0000","last_became_peered":"2023-11-08T08:21:31.702806+0000","last_unstale":"2023-11-08T08:21:33.099583+0000","last_undegraded":"2023-11-08T08:21:33.099583+0000","last_fullsized":"2023-11-08T08:21:33.099583+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:53:59.457113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291441+0000","last_change":"2023-11-08T08:21:32.098934+0000","last_active":"2023-11-08T08:21:33.291441+0000","last_peered":"2023-11-08T08:21:33.291441+0000","last_clean":"2023-11-08T08:21:33.291441+0000","last_became_active":"2023-11-08T08:21:32.098662+0000","last_became_peered":"2023-11-08T08:21:32.098662+0000","last_unstale":"2023-11-08T08:21:33.291441+0000","last_undegraded":"2023-11-08T08:21:33.291441+0000","last_fullsized":"2023-11-08T08:21:33.291441+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:26:04.016526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1c","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290897+0000","last_change":"2023-11-08T08:21:31.702201+0000","last_active":"2023-11-08T08:21:33.290897+0000","last_peered":"2023-11-08T08:21:33.290897+0000","last_clean":"2023-11-08T08:21:33.290897+0000","last_became_active":"2023-11-08T08:21:31.701923+0000","last_became_peered":"2023-11-08T08:21:31.701923+0000","last_unstale":"2023-11-08T08:21:33.290897+0000","last_undegraded":"2023-11-08T08:21:33.290897+0000","last_fullsized":"2023-11-08T08:21:33.290897+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:28:32.357569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.175803+0000","last_change":"2023-11-08T08:21:32.104074+0000","last_active":"2023-11-08T08:21:33.175803+0000","last_peered":"2023-11-08T08:21:33.175803+0000","last_clean":"2023-11-08T08:21:33.175803+0000","last_became_active":"2023-11-08T08:21:32.103795+0000","last_became_peered":"2023-11-08T08:21:32.103795+0000","last_unstale":"2023-11-08T08:21:33.175803+0000","last_undegraded":"2023-11-08T08:21:33.175803+0000","last_fullsized":"2023-11-08T08:21:33.175803+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:38:20.555180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176213+0000","last_change":"2023-11-08T08:21:31.696909+0000","last_active":"2023-11-08T08:21:33.176213+0000","last_peered":"2023-11-08T08:21:33.176213+0000","last_clean":"2023-11-08T08:21:33.176213+0000","last_became_active":"2023-11-08T08:21:31.696589+0000","last_became_peered":"2023-11-08T08:21:31.696589+0000","last_unstale":"2023-11-08T08:21:33.176213+0000","last_undegraded":"2023-11-08T08:21:33.176213+0000","last_fullsized":"2023-11-08T08:21:33.176213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:32:15.740195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098929+0000","last_change":"2023-11-08T08:21:32.105999+0000","last_active":"2023-11-08T08:21:33.098929+0000","last_peered":"2023-11-08T08:21:33.098929+0000","last_clean":"2023-11-08T08:21:33.098929+0000","last_became_active":"2023-11-08T08:21:32.102873+0000","last_became_peered":"2023-11-08T08:21:32.102873+0000","last_unstale":"2023-11-08T08:21:33.098929+0000","last_undegraded":"2023-11-08T08:21:33.098929+0000","last_fullsized":"2023-11-08T08:21:33.098929+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:28:54.922372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1a","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290808+0000","last_change":"2023-11-08T08:21:31.703524+0000","last_active":"2023-11-08T08:21:33.290808+0000","last_peered":"2023-11-08T08:21:33.290808+0000","last_clean":"2023-11-08T08:21:33.290808+0000","last_became_active":"2023-11-08T08:21:31.703256+0000","last_became_peered":"2023-11-08T08:21:31.703256+0000","last_unstale":"2023-11-08T08:21:33.290808+0000","last_undegraded":"2023-11-08T08:21:33.290808+0000","last_fullsized":"2023-11-08T08:21:33.290808+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:59:18.212331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098415+0000","last_change":"2023-11-08T08:21:32.101609+0000","last_active":"2023-11-08T08:21:33.098415+0000","last_peered":"2023-11-08T08:21:33.098415+0000","last_clean":"2023-11-08T08:21:33.098415+0000","last_became_active":"2023-11-08T08:21:32.101327+0000","last_became_peered":"2023-11-08T08:21:32.101327+0000","last_unstale":"2023-11-08T08:21:33.098415+0000","last_undegraded":"2023-11-08T08:21:33.098415+0000","last_fullsized":"2023-11-08T08:21:33.098415+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:19:17.115635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099238+0000","last_change":"2023-11-08T08:21:31.707009+0000","last_active":"2023-11-08T08:21:33.099238+0000","last_peered":"2023-11-08T08:21:33.099238+0000","last_clean":"2023-11-08T08:21:33.099238+0000","last_became_active":"2023-11-08T08:21:31.706743+0000","last_became_peered":"2023-11-08T08:21:31.706743+0000","last_unstale":"2023-11-08T08:21:33.099238+0000","last_undegraded":"2023-11-08T08:21:33.099238+0000","last_fullsized":"2023-11-08T08:21:33.099238+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:58.572397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.175967+0000","last_change":"2023-11-08T08:21:32.104046+0000","last_active":"2023-11-08T08:21:33.175967+0000","last_peered":"2023-11-08T08:21:33.175967+0000","last_clean":"2023-11-08T08:21:33.175967+0000","last_became_active":"2023-11-08T08:21:32.100653+0000","last_became_peered":"2023-11-08T08:21:32.100653+0000","last_unstale":"2023-11-08T08:21:33.175967+0000","last_undegraded":"2023-11-08T08:21:33.175967+0000","last_fullsized":"2023-11-08T08:21:33.175967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:26:25.345947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290824+0000","last_change":"2023-11-08T08:21:31.703174+0000","last_active":"2023-11-08T08:21:33.290824+0000","last_peered":"2023-11-08T08:21:33.290824+0000","last_clean":"2023-11-08T08:21:33.290824+0000","last_became_active":"2023-11-08T08:21:31.702884+0000","last_became_peered":"2023-11-08T08:21:31.702884+0000","last_unstale":"2023-11-08T08:21:33.290824+0000","last_undegraded":"2023-11-08T08:21:33.290824+0000","last_fullsized":"2023-11-08T08:21:33.290824+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:14:00.278020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099494+0000","last_change":"2023-11-08T08:21:32.105793+0000","last_active":"2023-11-08T08:21:33.099494+0000","last_peered":"2023-11-08T08:21:33.099494+0000","last_clean":"2023-11-08T08:21:33.099494+0000","last_became_active":"2023-11-08T08:21:32.105456+0000","last_became_peered":"2023-11-08T08:21:32.105456+0000","last_unstale":"2023-11-08T08:21:33.099494+0000","last_undegraded":"2023-11-08T08:21:33.099494+0000","last_fullsized":"2023-11-08T08:21:33.099494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:56:19.112133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291392+0000","last_change":"2023-11-08T08:21:31.707361+0000","last_active":"2023-11-08T08:21:33.291392+0000","last_peered":"2023-11-08T08:21:33.291392+0000","last_clean":"2023-11-08T08:21:33.291392+0000","last_became_active":"2023-11-08T08:21:31.707052+0000","last_became_peered":"2023-11-08T08:21:31.707052+0000","last_unstale":"2023-11-08T08:21:33.291392+0000","last_undegraded":"2023-11-08T08:21:33.291392+0000","last_fullsized":"2023-11-08T08:21:33.291392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:50.904536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176171+0000","last_change":"2023-11-08T08:21:32.106778+0000","last_active":"2023-11-08T08:21:33.176171+0000","last_peered":"2023-11-08T08:21:33.176171+0000","last_clean":"2023-11-08T08:21:33.176171+0000","last_became_active":"2023-11-08T08:21:32.106444+0000","last_became_peered":"2023-11-08T08:21:32.106444+0000","last_unstale":"2023-11-08T08:21:33.176171+0000","last_undegraded":"2023-11-08T08:21:33.176171+0000","last_fullsized":"2023-11-08T08:21:33.176171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:38:37.762000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096765+0000","last_change":"2023-11-08T08:21:31.704456+0000","last_active":"2023-11-08T08:21:33.096765+0000","last_peered":"2023-11-08T08:21:33.096765+0000","last_clean":"2023-11-08T08:21:33.096765+0000","last_became_active":"2023-11-08T08:21:31.704190+0000","last_became_peered":"2023-11-08T08:21:31.704190+0000","last_unstale":"2023-11-08T08:21:33.096765+0000","last_undegraded":"2023-11-08T08:21:33.096765+0000","last_fullsized":"2023-11-08T08:21:33.096765+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:36:54.332089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096597+0000","last_change":"2023-11-08T08:21:32.100162+0000","last_active":"2023-11-08T08:21:33.096597+0000","last_peered":"2023-11-08T08:21:33.096597+0000","last_clean":"2023-11-08T08:21:33.096597+0000","last_became_active":"2023-11-08T08:21:32.099890+0000","last_became_peered":"2023-11-08T08:21:32.099890+0000","last_unstale":"2023-11-08T08:21:33.096597+0000","last_undegraded":"2023-11-08T08:21:33.096597+0000","last_fullsized":"2023-11-08T08:21:33.096597+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:36:16.410656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.15","version":"55'3","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290557+0000","last_change":"2023-11-08T08:21:31.709724+0000","last_active":"2023-11-08T08:21:33.290557+0000","last_peered":"2023-11-08T08:21:33.290557+0000","last_clean":"2023-11-08T08:21:33.290557+0000","last_became_active":"2023-11-08T08:21:31.709428+0000","last_became_peered":"2023-11-08T08:21:31.709428+0000","last_unstale":"2023-11-08T08:21:33.290557+0000","last_undegraded":"2023-11-08T08:21:33.290557+0000","last_fullsized":"2023-11-08T08:21:33.290557+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:26:03.008514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291171+0000","last_change":"2023-11-08T08:21:32.101170+0000","last_active":"2023-11-08T08:21:33.291171+0000","last_peered":"2023-11-08T08:21:33.291171+0000","last_clean":"2023-11-08T08:21:33.291171+0000","last_became_active":"2023-11-08T08:21:32.100904+0000","last_became_peered":"2023-11-08T08:21:32.100904+0000","last_unstale":"2023-11-08T08:21:33.291171+0000","last_undegraded":"2023-11-08T08:21:33.291171+0000","last_fullsized":"2023-11-08T08:21:33.291171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:58:07.081698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"55'6","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290998+0000","last_change":"2023-11-08T08:21:31.700460+0000","last_active":"2023-11-08T08:21:33.290998+0000","last_peered":"2023-11-08T08:21:33.290998+0000","last_clean":"2023-11-08T08:21:33.290998+0000","last_became_active":"2023-11-08T08:21:31.700153+0000","last_became_peered":"2023-11-08T08:21:31.700153+0000","last_unstale":"2023-11-08T08:21:33.290998+0000","last_undegraded":"2023-11-08T08:21:33.290998+0000","last_fullsized":"2023-11-08T08:21:33.290998+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:04:51.874801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098231+0000","last_change":"2023-11-08T08:21:32.106213+0000","last_active":"2023-11-08T08:21:33.098231+0000","last_peered":"2023-11-08T08:21:33.098231+0000","last_clean":"2023-11-08T08:21:33.098231+0000","last_became_active":"2023-11-08T08:21:32.105962+0000","last_became_peered":"2023-11-08T08:21:32.105962+0000","last_unstale":"2023-11-08T08:21:33.098231+0000","last_undegraded":"2023-11-08T08:21:33.098231+0000","last_fullsized":"2023-11-08T08:21:33.098231+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:44:47.641703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291230+0000","last_change":"2023-11-08T08:21:31.704944+0000","last_active":"2023-11-08T08:21:33.291230+0000","last_peered":"2023-11-08T08:21:33.291230+0000","last_clean":"2023-11-08T08:21:33.291230+0000","last_became_active":"2023-11-08T08:21:31.703990+0000","last_became_peered":"2023-11-08T08:21:31.703990+0000","last_unstale":"2023-11-08T08:21:33.291230+0000","last_undegraded":"2023-11-08T08:21:33.291230+0000","last_fullsized":"2023-11-08T08:21:33.291230+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:24:33.396545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098351+0000","last_change":"2023-11-08T08:21:32.105400+0000","last_active":"2023-11-08T08:21:33.098351+0000","last_peered":"2023-11-08T08:21:33.098351+0000","last_clean":"2023-11-08T08:21:33.098351+0000","last_became_active":"2023-11-08T08:21:32.105143+0000","last_became_peered":"2023-11-08T08:21:32.105143+0000","last_unstale":"2023-11-08T08:21:33.098351+0000","last_undegraded":"2023-11-08T08:21:33.098351+0000","last_fullsized":"2023-11-08T08:21:33.098351+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:49:49.217341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098437+0000","last_change":"2023-11-08T08:21:31.699946+0000","last_active":"2023-11-08T08:21:33.098437+0000","last_peered":"2023-11-08T08:21:33.098437+0000","last_clean":"2023-11-08T08:21:33.098437+0000","last_became_active":"2023-11-08T08:21:31.699463+0000","last_became_peered":"2023-11-08T08:21:31.699463+0000","last_unstale":"2023-11-08T08:21:33.098437+0000","last_undegraded":"2023-11-08T08:21:33.098437+0000","last_fullsized":"2023-11-08T08:21:33.098437+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:31.230400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098526+0000","last_change":"2023-11-08T08:21:32.107991+0000","last_active":"2023-11-08T08:21:33.098526+0000","last_peered":"2023-11-08T08:21:33.098526+0000","last_clean":"2023-11-08T08:21:33.098526+0000","last_became_active":"2023-11-08T08:21:32.107723+0000","last_became_peered":"2023-11-08T08:21:32.107723+0000","last_unstale":"2023-11-08T08:21:33.098526+0000","last_undegraded":"2023-11-08T08:21:33.098526+0000","last_fullsized":"2023-11-08T08:21:33.098526+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:53:35.233536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290387+0000","last_change":"2023-11-08T08:21:31.703510+0000","last_active":"2023-11-08T08:21:33.290387+0000","last_peered":"2023-11-08T08:21:33.290387+0000","last_clean":"2023-11-08T08:21:33.290387+0000","last_became_active":"2023-11-08T08:21:31.703226+0000","last_became_peered":"2023-11-08T08:21:31.703226+0000","last_unstale":"2023-11-08T08:21:33.290387+0000","last_undegraded":"2023-11-08T08:21:33.290387+0000","last_fullsized":"2023-11-08T08:21:33.290387+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:19:16.107623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099017+0000","last_change":"2023-11-08T08:21:32.107787+0000","last_active":"2023-11-08T08:21:33.099017+0000","last_peered":"2023-11-08T08:21:33.099017+0000","last_clean":"2023-11-08T08:21:33.099017+0000","last_became_active":"2023-11-08T08:21:32.107456+0000","last_became_peered":"2023-11-08T08:21:32.107456+0000","last_unstale":"2023-11-08T08:21:33.099017+0000","last_undegraded":"2023-11-08T08:21:33.099017+0000","last_fullsized":"2023-11-08T08:21:33.099017+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:44:10.321033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"55'8","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098250+0000","last_change":"2023-11-08T08:21:31.697573+0000","last_active":"2023-11-08T08:21:33.098250+0000","last_peered":"2023-11-08T08:21:33.098250+0000","last_clean":"2023-11-08T08:21:33.098250+0000","last_became_active":"2023-11-08T08:21:31.697410+0000","last_became_peered":"2023-11-08T08:21:31.697410+0000","last_unstale":"2023-11-08T08:21:33.098250+0000","last_undegraded":"2023-11-08T08:21:33.098250+0000","last_fullsized":"2023-11-08T08:21:33.098250+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:56:18.104122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176210+0000","last_change":"2023-11-08T08:21:32.110171+0000","last_active":"2023-11-08T08:21:33.176210+0000","last_peered":"2023-11-08T08:21:33.176210+0000","last_clean":"2023-11-08T08:21:33.176210+0000","last_became_active":"2023-11-08T08:21:32.109871+0000","last_became_peered":"2023-11-08T08:21:32.109871+0000","last_unstale":"2023-11-08T08:21:33.176210+0000","last_undegraded":"2023-11-08T08:21:33.176210+0000","last_fullsized":"2023-11-08T08:21:33.176210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:58:07.081698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291444+0000","last_change":"2023-11-08T08:21:31.708438+0000","last_active":"2023-11-08T08:21:33.291444+0000","last_peered":"2023-11-08T08:21:33.291444+0000","last_clean":"2023-11-08T08:21:33.291444+0000","last_became_active":"2023-11-08T08:21:31.708206+0000","last_became_peered":"2023-11-08T08:21:31.708206+0000","last_unstale":"2023-11-08T08:21:33.291444+0000","last_undegraded":"2023-11-08T08:21:33.291444+0000","last_fullsized":"2023-11-08T08:21:33.291444+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:51:09.105977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176135+0000","last_change":"2023-11-08T08:21:32.112116+0000","last_active":"2023-11-08T08:21:33.176135+0000","last_peered":"2023-11-08T08:21:33.176135+0000","last_clean":"2023-11-08T08:21:33.176135+0000","last_became_active":"2023-11-08T08:21:32.111476+0000","last_became_peered":"2023-11-08T08:21:32.111476+0000","last_unstale":"2023-11-08T08:21:33.176135+0000","last_undegraded":"2023-11-08T08:21:33.176135+0000","last_fullsized":"2023-11-08T08:21:33.176135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:57:30.530999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291394+0000","last_change":"2023-11-08T08:21:31.705424+0000","last_active":"2023-11-08T08:21:33.291394+0000","last_peered":"2023-11-08T08:21:33.291394+0000","last_clean":"2023-11-08T08:21:33.291394+0000","last_became_active":"2023-11-08T08:21:31.704184+0000","last_became_peered":"2023-11-08T08:21:31.704184+0000","last_unstale":"2023-11-08T08:21:33.291394+0000","last_undegraded":"2023-11-08T08:21:33.291394+0000","last_fullsized":"2023-11-08T08:21:33.291394+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:25:14.455165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290507+0000","last_change":"2023-11-08T08:21:32.104411+0000","last_active":"2023-11-08T08:21:33.290507+0000","last_peered":"2023-11-08T08:21:33.290507+0000","last_clean":"2023-11-08T08:21:33.290507+0000","last_became_active":"2023-11-08T08:21:32.104130+0000","last_became_peered":"2023-11-08T08:21:32.104130+0000","last_unstale":"2023-11-08T08:21:33.290507+0000","last_undegraded":"2023-11-08T08:21:33.290507+0000","last_fullsized":"2023-11-08T08:21:33.290507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:50:43.841421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290924+0000","last_change":"2023-11-08T08:21:31.700555+0000","last_active":"2023-11-08T08:21:33.290924+0000","last_peered":"2023-11-08T08:21:33.290924+0000","last_clean":"2023-11-08T08:21:33.290924+0000","last_became_active":"2023-11-08T08:21:31.699947+0000","last_became_peered":"2023-11-08T08:21:31.699947+0000","last_unstale":"2023-11-08T08:21:33.290924+0000","last_undegraded":"2023-11-08T08:21:33.290924+0000","last_fullsized":"2023-11-08T08:21:33.290924+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:36:15.402645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290777+0000","last_change":"2023-11-08T08:21:32.098843+0000","last_active":"2023-11-08T08:21:33.290777+0000","last_peered":"2023-11-08T08:21:33.290777+0000","last_clean":"2023-11-08T08:21:33.290777+0000","last_became_active":"2023-11-08T08:21:32.098583+0000","last_became_peered":"2023-11-08T08:21:32.098583+0000","last_unstale":"2023-11-08T08:21:33.290777+0000","last_undegraded":"2023-11-08T08:21:33.290777+0000","last_fullsized":"2023-11-08T08:21:33.290777+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:38:37.762000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291098+0000","last_change":"2023-11-08T08:21:31.706776+0000","last_active":"2023-11-08T08:21:33.291098+0000","last_peered":"2023-11-08T08:21:33.291098+0000","last_clean":"2023-11-08T08:21:33.291098+0000","last_became_active":"2023-11-08T08:21:31.706439+0000","last_became_peered":"2023-11-08T08:21:31.706439+0000","last_unstale":"2023-11-08T08:21:33.291098+0000","last_undegraded":"2023-11-08T08:21:33.291098+0000","last_fullsized":"2023-11-08T08:21:33.291098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:30:20.916773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290810+0000","last_change":"2023-11-08T08:21:32.108904+0000","last_active":"2023-11-08T08:21:33.290810+0000","last_peered":"2023-11-08T08:21:33.290810+0000","last_clean":"2023-11-08T08:21:33.290810+0000","last_became_active":"2023-11-08T08:21:32.108655+0000","last_became_peered":"2023-11-08T08:21:32.108655+0000","last_unstale":"2023-11-08T08:21:33.290810+0000","last_undegraded":"2023-11-08T08:21:33.290810+0000","last_fullsized":"2023-11-08T08:21:33.290810+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:26.497082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096462+0000","last_change":"2023-11-08T08:21:31.702419+0000","last_active":"2023-11-08T08:21:33.096462+0000","last_peered":"2023-11-08T08:21:33.096462+0000","last_clean":"2023-11-08T08:21:33.096462+0000","last_became_active":"2023-11-08T08:21:31.702072+0000","last_became_peered":"2023-11-08T08:21:31.702072+0000","last_unstale":"2023-11-08T08:21:33.096462+0000","last_undegraded":"2023-11-08T08:21:33.096462+0000","last_fullsized":"2023-11-08T08:21:33.096462+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:09:21.883003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096382+0000","last_change":"2023-11-08T08:21:32.100545+0000","last_active":"2023-11-08T08:21:33.096382+0000","last_peered":"2023-11-08T08:21:33.096382+0000","last_clean":"2023-11-08T08:21:33.096382+0000","last_became_active":"2023-11-08T08:21:32.099949+0000","last_became_peered":"2023-11-08T08:21:32.099949+0000","last_unstale":"2023-11-08T08:21:33.096382+0000","last_undegraded":"2023-11-08T08:21:33.096382+0000","last_fullsized":"2023-11-08T08:21:33.096382+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:59.580409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098969+0000","last_change":"2023-11-08T08:21:31.705077+0000","last_active":"2023-11-08T08:21:33.098969+0000","last_peered":"2023-11-08T08:21:33.098969+0000","last_clean":"2023-11-08T08:21:33.098969+0000","last_became_active":"2023-11-08T08:21:31.704297+0000","last_became_peered":"2023-11-08T08:21:31.704297+0000","last_unstale":"2023-11-08T08:21:33.098969+0000","last_undegraded":"2023-11-08T08:21:33.098969+0000","last_fullsized":"2023-11-08T08:21:33.098969+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:15.664229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290840+0000","last_change":"2023-11-08T08:21:32.102465+0000","last_active":"2023-11-08T08:21:33.290840+0000","last_peered":"2023-11-08T08:21:33.290840+0000","last_clean":"2023-11-08T08:21:33.290840+0000","last_became_active":"2023-11-08T08:21:32.102157+0000","last_became_peered":"2023-11-08T08:21:32.102157+0000","last_unstale":"2023-11-08T08:21:33.290840+0000","last_undegraded":"2023-11-08T08:21:33.290840+0000","last_fullsized":"2023-11-08T08:21:33.290840+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:28:54.922372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290805+0000","last_change":"2023-11-08T08:21:31.703438+0000","last_active":"2023-11-08T08:21:33.290805+0000","last_peered":"2023-11-08T08:21:33.290805+0000","last_clean":"2023-11-08T08:21:33.290805+0000","last_became_active":"2023-11-08T08:21:31.703152+0000","last_became_peered":"2023-11-08T08:21:31.703152+0000","last_unstale":"2023-11-08T08:21:33.290805+0000","last_undegraded":"2023-11-08T08:21:33.290805+0000","last_fullsized":"2023-11-08T08:21:33.290805+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:26:32.249721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176103+0000","last_change":"2023-11-08T08:21:32.100884+0000","last_active":"2023-11-08T08:21:33.176103+0000","last_peered":"2023-11-08T08:21:33.176103+0000","last_clean":"2023-11-08T08:21:33.176103+0000","last_became_active":"2023-11-08T08:21:32.100292+0000","last_became_peered":"2023-11-08T08:21:32.100292+0000","last_unstale":"2023-11-08T08:21:33.176103+0000","last_undegraded":"2023-11-08T08:21:33.176103+0000","last_fullsized":"2023-11-08T08:21:33.176103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:31.579603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290786+0000","last_change":"2023-11-08T08:21:31.705118+0000","last_active":"2023-11-08T08:21:33.290786+0000","last_peered":"2023-11-08T08:21:33.290786+0000","last_clean":"2023-11-08T08:21:33.290786+0000","last_became_active":"2023-11-08T08:21:31.704853+0000","last_became_peered":"2023-11-08T08:21:31.704853+0000","last_unstale":"2023-11-08T08:21:33.290786+0000","last_undegraded":"2023-11-08T08:21:33.290786+0000","last_fullsized":"2023-11-08T08:21:33.290786+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:00:02.693157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096687+0000","last_change":"2023-11-08T08:21:32.104693+0000","last_active":"2023-11-08T08:21:33.096687+0000","last_peered":"2023-11-08T08:21:33.096687+0000","last_clean":"2023-11-08T08:21:33.096687+0000","last_became_active":"2023-11-08T08:21:32.104445+0000","last_became_peered":"2023-11-08T08:21:32.104445+0000","last_unstale":"2023-11-08T08:21:33.096687+0000","last_undegraded":"2023-11-08T08:21:33.096687+0000","last_fullsized":"2023-11-08T08:21:33.096687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:51.912547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290794+0000","last_change":"2023-11-08T08:21:31.703194+0000","last_active":"2023-11-08T08:21:33.290794+0000","last_peered":"2023-11-08T08:21:33.290794+0000","last_clean":"2023-11-08T08:21:33.290794+0000","last_became_active":"2023-11-08T08:21:31.702802+0000","last_became_peered":"2023-11-08T08:21:31.702802+0000","last_unstale":"2023-11-08T08:21:33.290794+0000","last_undegraded":"2023-11-08T08:21:33.290794+0000","last_fullsized":"2023-11-08T08:21:33.290794+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:43:30.196976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.095809+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:21:32.095809+0000","last_peered":"2023-11-08T08:21:32.095809+0000","last_clean":"2023-11-08T08:21:32.095809+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:32.095809+0000","last_undegraded":"2023-11-08T08:21:32.095809+0000","last_fullsized":"2023-11-08T08:21:32.095809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290686+0000","last_change":"2023-11-08T08:21:32.105692+0000","last_active":"2023-11-08T08:21:33.290686+0000","last_peered":"2023-11-08T08:21:33.290686+0000","last_clean":"2023-11-08T08:21:33.290686+0000","last_became_active":"2023-11-08T08:21:32.105425+0000","last_became_peered":"2023-11-08T08:21:32.105425+0000","last_unstale":"2023-11-08T08:21:33.290686+0000","last_undegraded":"2023-11-08T08:21:33.290686+0000","last_fullsized":"2023-11-08T08:21:33.290686+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:21.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291167+0000","last_change":"2023-11-08T08:21:31.697385+0000","last_active":"2023-11-08T08:21:33.291167+0000","last_peered":"2023-11-08T08:21:33.291167+0000","last_clean":"2023-11-08T08:21:33.291167+0000","last_became_active":"2023-11-08T08:21:31.697071+0000","last_became_peered":"2023-11-08T08:21:31.697071+0000","last_unstale":"2023-11-08T08:21:33.291167+0000","last_undegraded":"2023-11-08T08:21:33.291167+0000","last_fullsized":"2023-11-08T08:21:33.291167+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:14:00.278020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":42,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.089841+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:21:32.089841+0000","last_peered":"2023-11-08T08:21:32.089841+0000","last_clean":"2023-11-08T08:21:32.089841+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:21:32.089841+0000","last_undegraded":"2023-11-08T08:21:32.089841+0000","last_fullsized":"2023-11-08T08:21:32.089841+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290891+0000","last_change":"2023-11-08T08:21:32.106974+0000","last_active":"2023-11-08T08:21:33.290891+0000","last_peered":"2023-11-08T08:21:33.290891+0000","last_clean":"2023-11-08T08:21:33.290891+0000","last_became_active":"2023-11-08T08:21:32.106728+0000","last_became_peered":"2023-11-08T08:21:32.106728+0000","last_unstale":"2023-11-08T08:21:33.290891+0000","last_undegraded":"2023-11-08T08:21:33.290891+0000","last_fullsized":"2023-11-08T08:21:33.290891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:44:15.097568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.b","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291086+0000","last_change":"2023-11-08T08:21:31.704398+0000","last_active":"2023-11-08T08:21:33.291086+0000","last_peered":"2023-11-08T08:21:33.291086+0000","last_clean":"2023-11-08T08:21:33.291086+0000","last_became_active":"2023-11-08T08:21:31.704087+0000","last_became_peered":"2023-11-08T08:21:31.704087+0000","last_unstale":"2023-11-08T08:21:33.291086+0000","last_undegraded":"2023-11-08T08:21:33.291086+0000","last_fullsized":"2023-11-08T08:21:33.291086+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:50.904536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.090168+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:21:32.090168+0000","last_peered":"2023-11-08T08:21:32.090168+0000","last_clean":"2023-11-08T08:21:32.090168+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:21:32.090168+0000","last_undegraded":"2023-11-08T08:21:32.090168+0000","last_fullsized":"2023-11-08T08:21:32.090168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":198,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.089992+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:21:32.089992+0000","last_peered":"2023-11-08T08:21:32.089992+0000","last_clean":"2023-11-08T08:21:32.089992+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:21:32.089992+0000","last_undegraded":"2023-11-08T08:21:32.089992+0000","last_fullsized":"2023-11-08T08:21:32.089992+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096243+0000","last_change":"2023-11-08T08:21:32.097008+0000","last_active":"2023-11-08T08:21:33.096243+0000","last_peered":"2023-11-08T08:21:33.096243+0000","last_clean":"2023-11-08T08:21:33.096243+0000","last_became_active":"2023-11-08T08:21:32.096755+0000","last_became_peered":"2023-11-08T08:21:32.096755+0000","last_unstale":"2023-11-08T08:21:33.096243+0000","last_undegraded":"2023-11-08T08:21:33.096243+0000","last_fullsized":"2023-11-08T08:21:33.096243+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:59:19.220343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098522+0000","last_change":"2023-11-08T08:21:31.703727+0000","last_active":"2023-11-08T08:21:33.098522+0000","last_peered":"2023-11-08T08:21:33.098522+0000","last_clean":"2023-11-08T08:21:33.098522+0000","last_became_active":"2023-11-08T08:21:31.703434+0000","last_became_peered":"2023-11-08T08:21:31.703434+0000","last_unstale":"2023-11-08T08:21:33.098522+0000","last_undegraded":"2023-11-08T08:21:33.098522+0000","last_fullsized":"2023-11-08T08:21:33.098522+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:25:16.991936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.097258+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:21:32.097258+0000","last_peered":"2023-11-08T08:21:32.097258+0000","last_clean":"2023-11-08T08:21:32.097258+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:21:32.097258+0000","last_undegraded":"2023-11-08T08:21:32.097258+0000","last_fullsized":"2023-11-08T08:21:32.097258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098443+0000","last_change":"2023-11-08T08:21:32.111444+0000","last_active":"2023-11-08T08:21:33.098443+0000","last_peered":"2023-11-08T08:21:33.098443+0000","last_clean":"2023-11-08T08:21:33.098443+0000","last_became_active":"2023-11-08T08:21:32.111181+0000","last_became_peered":"2023-11-08T08:21:32.111181+0000","last_unstale":"2023-11-08T08:21:33.098443+0000","last_undegraded":"2023-11-08T08:21:33.098443+0000","last_fullsized":"2023-11-08T08:21:33.098443+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:26:04.016526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290515+0000","last_change":"2023-11-08T08:21:31.702934+0000","last_active":"2023-11-08T08:21:33.290515+0000","last_peered":"2023-11-08T08:21:33.290515+0000","last_clean":"2023-11-08T08:21:33.290515+0000","last_became_active":"2023-11-08T08:21:31.702659+0000","last_became_peered":"2023-11-08T08:21:31.702659+0000","last_unstale":"2023-11-08T08:21:33.290515+0000","last_undegraded":"2023-11-08T08:21:33.290515+0000","last_fullsized":"2023-11-08T08:21:33.290515+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:15.664229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.092696+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:21:32.092696+0000","last_peered":"2023-11-08T08:21:32.092696+0000","last_clean":"2023-11-08T08:21:32.092696+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:32.092696+0000","last_undegraded":"2023-11-08T08:21:32.092696+0000","last_fullsized":"2023-11-08T08:21:32.092696+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291068+0000","last_change":"2023-11-08T08:21:32.097112+0000","last_active":"2023-11-08T08:21:33.291068+0000","last_peered":"2023-11-08T08:21:33.291068+0000","last_clean":"2023-11-08T08:21:33.291068+0000","last_became_active":"2023-11-08T08:21:32.096382+0000","last_became_peered":"2023-11-08T08:21:32.096382+0000","last_unstale":"2023-11-08T08:21:33.291068+0000","last_undegraded":"2023-11-08T08:21:33.291068+0000","last_fullsized":"2023-11-08T08:21:33.291068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:11:14.299828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290999+0000","last_change":"2023-11-08T08:21:31.705761+0000","last_active":"2023-11-08T08:21:33.290999+0000","last_peered":"2023-11-08T08:21:33.290999+0000","last_clean":"2023-11-08T08:21:33.290999+0000","last_became_active":"2023-11-08T08:21:31.705490+0000","last_became_peered":"2023-11-08T08:21:31.705490+0000","last_unstale":"2023-11-08T08:21:33.290999+0000","last_undegraded":"2023-11-08T08:21:33.290999+0000","last_fullsized":"2023-11-08T08:21:33.290999+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:51:09.105977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.088808+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:21:32.088808+0000","last_peered":"2023-11-08T08:21:32.088808+0000","last_clean":"2023-11-08T08:21:32.088808+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:21:32.088808+0000","last_undegraded":"2023-11-08T08:21:32.088808+0000","last_fullsized":"2023-11-08T08:21:32.088808+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096504+0000","last_change":"2023-11-08T08:21:32.099734+0000","last_active":"2023-11-08T08:21:33.096504+0000","last_peered":"2023-11-08T08:21:33.096504+0000","last_clean":"2023-11-08T08:21:33.096504+0000","last_became_active":"2023-11-08T08:21:32.099461+0000","last_became_peered":"2023-11-08T08:21:32.099461+0000","last_unstale":"2023-11-08T08:21:33.096504+0000","last_undegraded":"2023-11-08T08:21:33.096504+0000","last_fullsized":"2023-11-08T08:21:33.096504+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:04:52.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291264+0000","last_change":"2023-11-08T08:21:31.705926+0000","last_active":"2023-11-08T08:21:33.291264+0000","last_peered":"2023-11-08T08:21:33.291264+0000","last_clean":"2023-11-08T08:21:33.291264+0000","last_became_active":"2023-11-08T08:21:31.705693+0000","last_became_peered":"2023-11-08T08:21:31.705693+0000","last_unstale":"2023-11-08T08:21:33.291264+0000","last_undegraded":"2023-11-08T08:21:33.291264+0000","last_fullsized":"2023-11-08T08:21:33.291264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:58.572397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.089337+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:21:32.089337+0000","last_peered":"2023-11-08T08:21:32.089337+0000","last_clean":"2023-11-08T08:21:32.089337+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:21:32.089337+0000","last_undegraded":"2023-11-08T08:21:32.089337+0000","last_fullsized":"2023-11-08T08:21:32.089337+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291131+0000","last_change":"2023-11-08T08:21:32.109819+0000","last_active":"2023-11-08T08:21:33.291131+0000","last_peered":"2023-11-08T08:21:33.291131+0000","last_clean":"2023-11-08T08:21:33.291131+0000","last_became_active":"2023-11-08T08:21:32.109510+0000","last_became_peered":"2023-11-08T08:21:32.109510+0000","last_unstale":"2023-11-08T08:21:33.291131+0000","last_undegraded":"2023-11-08T08:21:33.291131+0000","last_fullsized":"2023-11-08T08:21:33.291131+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:39:47.423926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291499+0000","last_change":"2023-11-08T08:21:31.707415+0000","last_active":"2023-11-08T08:21:33.291499+0000","last_peered":"2023-11-08T08:21:33.291499+0000","last_clean":"2023-11-08T08:21:33.291499+0000","last_became_active":"2023-11-08T08:21:31.706556+0000","last_became_peered":"2023-11-08T08:21:31.706556+0000","last_unstale":"2023-11-08T08:21:33.291499+0000","last_undegraded":"2023-11-08T08:21:33.291499+0000","last_fullsized":"2023-11-08T08:21:33.291499+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:15.664229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.092202+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:21:32.092202+0000","last_peered":"2023-11-08T08:21:32.092202+0000","last_clean":"2023-11-08T08:21:32.092202+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:21:32.092202+0000","last_undegraded":"2023-11-08T08:21:32.092202+0000","last_fullsized":"2023-11-08T08:21:32.092202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9240,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362600,"statfs":{"total":96636764160,"available":96627302400,"internally_reserved":0,"allocated":671744,"data_stored":292689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63500000000000001,"5min":1.611,"15min":1.611},"last":0.55600000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.71199999999999997,"15min":0.73799999999999999},"min":{"1min":0.33500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.92500000000000004,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70999999999999996,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.501}]},{"osd":2,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.42099999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.98399999999999999,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.438},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.78400000000000003,"15min":0.81899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.73499999999999999,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.57699999999999996}]},{"osd":3,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.45800000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.58899999999999997,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.59199999999999997,"15min":0.60799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71199999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48599999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.623,"5min":0.872,"15min":0.872},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.63300000000000001,"15min":0.64800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.66600000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59599999999999997}]},{"osd":6,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.65500000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.748,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9384,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362456,"statfs":{"total":96636764160,"available":96627154944,"internally_reserved":0,"allocated":688128,"data_stored":293612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66300000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57199999999999995,"5min":1.115,"15min":1.115},"last":0.82499999999999996}]},{"osd":1,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69499999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.38300000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71699999999999997,"5min":1.089,"15min":1.089},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.64000000000000001,"5min":0.996,"15min":0.996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50600000000000001,"15min":0.502},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.65200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.629}]},{"osd":3,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.39100000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":1.135,"15min":1.135},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.44400000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69999999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72999999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.86099999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.39100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.84299999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.74399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9372,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362468,"statfs":{"total":96636764160,"available":96627167232,"internally_reserved":0,"allocated":675840,"data_stored":292723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.312,"15min":0.308},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.316,"15min":0.308},"min":{"1min":0.249,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.313}]},{"osd":2,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.216,"15min":0.216},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.312,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33600000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.378},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.37,"15min":0.35599999999999998},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.36099999999999999}]},{"osd":4,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.61599999999999999,"15min":0.56699999999999995},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":9.0380000000000003,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":8.5169999999999995,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.496}]},{"osd":5,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":8.9979999999999993,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.46500000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.70999999999999996,"15min":0.67000000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":8.5860000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.67200000000000004,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.60799999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.67500000000000004,"15min":0.629},"min":{"1min":0.45800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":8.6790000000000003,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.69599999999999995,"15min":0.64800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":9.0850000000000009,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.55300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9304,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362536,"statfs":{"total":96636764160,"available":96627236864,"internally_reserved":0,"allocated":671744,"data_stored":292689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.48199999999999998,"15min":0.504},"min":{"1min":0.22900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.71199999999999997,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.50600000000000001,"15min":0.53000000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.69299999999999995,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.374}]},{"osd":2,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.749,"5min":2.032,"15min":2.032},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.42099999999999999,"15min":0.432},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66000000000000003,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.29399999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67300000000000004,"5min":1.788,"15min":1.788},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.311,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.77400000000000002,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.317}]},{"osd":4,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.379},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.47599999999999998}]},{"osd":5,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92300000000000004,"5min":1.825,"15min":1.825},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.627,"15min":0.625},"min":{"1min":0.42099999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.012,"5min":1.75,"15min":1.75},"last":0.51500000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.41599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.98799999999999999,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.58799999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.022,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9504,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362336,"statfs":{"total":96636764160,"available":96627032064,"internally_reserved":0,"allocated":679936,"data_stored":292745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.39500000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.309,"15min":0.309},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.379,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.27400000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.309,"15min":0.309},"min":{"1min":0.255,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.39900000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.377},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.307,"5min":0.22,"15min":0.22},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.46300000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.41799999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.52900000000000003},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.62},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.43099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.52000000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55000000000000004,"15min":0.54200000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.63700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10532,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361308,"statfs":{"total":96636764160,"available":96625979392,"internally_reserved":0,"allocated":1273856,"data_stored":883217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53200000000000003,"5min":1.319,"15min":1.319},"last":0.315},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.58899999999999997,"5min":1.915,"15min":1.915},"last":0.34699999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.41599999999999998,"15min":0.42199999999999999},"min":{"1min":0.25900000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.88300000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.39300000000000002,"15min":0.39900000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.71199999999999997,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65500000000000003,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.625,"5min":1.278,"15min":1.278},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.66300000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82399999999999995,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71599999999999997,"15min":0.72799999999999998},"min":{"1min":0.438,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.94499999999999995,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.41399999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96999999999999997,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":1.371,"15min":1.371},"last":0.64900000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70699999999999996,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89000000000000001,"5min":1.857,"15min":1.857},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.73999999999999999,"15min":0.753},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98499999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.71099999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67600000000000005,"15min":0.68600000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.67800000000000005},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86099999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94362084,"statfs":{"total":96636764160,"available":96626774016,"internally_reserved":0,"allocated":675840,"data_stored":293255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.42099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72799999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.47499999999999998,"15min":0.46800000000000003},"min":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83499999999999996,"5min":1.073,"15min":1.073},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70799999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.60499999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.57699999999999996},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.69999999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66000000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.625,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69999999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.55600000000000005}]},{"osd":7,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.63800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69699999999999995,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61699999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10544,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361296,"statfs":{"total":96636764160,"available":96625967104,"internally_reserved":0,"allocated":1286144,"data_stored":884717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.93000000000000005,"15min":0.996},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.92000000000000004,"15min":0.98199999999999998},"min":{"1min":0.45100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70299999999999996,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.86199999999999999,"15min":0.91000000000000003},"min":{"1min":0.48099999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72099999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.86299999999999999,"15min":0.90800000000000003},"min":{"1min":0.437,"5min":0.378,"15min":0.378},"max":{"1min":0.72199999999999998,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.86799999999999999,"15min":0.92300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69199999999999995,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.54200000000000004}]},{"osd":3,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96599999999999997,"15min":1.034},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77700000000000002,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.753,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.495},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36299999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.433,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.432}]},{"osd":7,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:21:36.541 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:21:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.692+0000 7f07ee98f700 1 Processor -- start 2023-11-08T08:21:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.692+0000 7f07ee98f700 1 -- start start 2023-11-08T08:21:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07ee98f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07e806e570 0x7f07e8160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07ee98f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80b1810 con 0x7f07e806a270 2023-11-08T08:21:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07ee98f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80b1af0 con 0x7f07e815cc80 2023-11-08T08:21:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07ee98f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e80b2490 con 0x7f07e806e570 2023-11-08T08:21:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07ecf2c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07e806e570 0x7f07e8160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07e7fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f07e806a270 0x7f07e806a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55364/0 (socket says 172.21.15.116:55364) 2023-11-08T08:21:36.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.693+0000 7f07e7fff700 1 -- 172.21.15.116:0/3617006885 learned_addr learned my addr 172.21.15.116:0/3617006885 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:36.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07ecf2c700 1 -- 172.21.15.116:0/3617006885 >> v1:172.21.15.129:6789/0 conn(0x7f07e815cc80 legacy=0x7f07e806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07ecf2c700 1 -- 172.21.15.116:0/3617006885 >> v1:172.21.15.116:6789/0 conn(0x7f07e806a270 legacy=0x7f07e806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07ecf2c700 1 -- 172.21.15.116:0/3617006885 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e80b1dd0 con 0x7f07e806e570 2023-11-08T08:21:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07e6ffd700 1 -- 172.21.15.116:0/3617006885 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052138227 0 0) 0x7f07e80b1810 con 0x7f07e806a270 2023-11-08T08:21:36.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07ecf2c700 1 --2- 172.21.15.116:0/3617006885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806e570 0x7f07e8160300 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009d90 tx=0x7f07e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10d04ec774b86ba server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07e6ffd700 1 -- 172.21.15.116:0/3617006885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e000e070 con 0x7f07e806e570 2023-11-08T08:21:36.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07e6ffd700 1 -- 172.21.15.116:0/3617006885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e000bb90 con 0x7f07e806e570 2023-11-08T08:21:36.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07e6ffd700 1 -- 172.21.15.116:0/3617006885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0003a20 con 0x7f07e806e570 2023-11-08T08:21:36.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07ee98f700 1 -- 172.21.15.116:0/3617006885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806e570 msgr2=0x7f07e8160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.694+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3617006885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806e570 0x7f07e8160300 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009d90 tx=0x7f07e000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.695+0000 7f07ee98f700 1 -- 172.21.15.116:0/3617006885 shutdown_connections 2023-11-08T08:21:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.695+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3617006885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806e570 0x7f07e8160300 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.695+0000 7f07ee98f700 1 -- 172.21.15.116:0/3617006885 >> 172.21.15.116:0/3617006885 conn(0x7f07e806c010 msgr2=0x7f07e80cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.695+0000 7f07ee98f700 1 -- 172.21.15.116:0/3617006885 shutdown_connections 2023-11-08T08:21:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.695+0000 7f07ee98f700 1 -- 172.21.15.116:0/3617006885 wait complete. 2023-11-08T08:21:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.696+0000 7f07ee98f700 1 Processor -- start 2023-11-08T08:21:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.696+0000 7f07ee98f700 1 -- start start 2023-11-08T08:21:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.696+0000 7f07ee98f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 0x7f07e80af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e806e570 0x7f07e81f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07e815cc80 0x7f07e81f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e81f5840 con 0x7f07e806a270 2023-11-08T08:21:36.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80af920 con 0x7f07e806e570 2023-11-08T08:21:36.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80afc00 con 0x7f07e815cc80 2023-11-08T08:21:36.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 0x7f07e80af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 0x7f07e80af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47820/0 (socket says 172.21.15.116:47820) 2023-11-08T08:21:36.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 -- 172.21.15.116:0/3505538012 learned_addr learned my addr 172.21.15.116:0/3505538012 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e77fe700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e806e570 0x7f07e81f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 -- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07e815cc80 msgr2=0x7f07e81f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ecf2c700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07e815cc80 0x7f07e81f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07e815cc80 0x7f07e81f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 -- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e806e570 msgr2=0x7f07e81f5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e806e570 0x7f07e81f5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 -- 172.21.15.116:0/3505538012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e0009a70 con 0x7f07e806a270 2023-11-08T08:21:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e77fe700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e806e570 0x7f07e81f5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ecf2c700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07e815cc80 0x7f07e81f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e7fff700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 0x7f07e80af290 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c520 tx=0x7f07d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8004c80 con 0x7f07e806a270 2023-11-08T08:21:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d8004e20 con 0x7f07e806a270 2023-11-08T08:21:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e815daa0 con 0x7f07e806a270 2023-11-08T08:21:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8007310 con 0x7f07e806a270 2023-11-08T08:21:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.697+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e80b20b0 con 0x7f07e806a270 2023-11-08T08:21:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.698+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e815dc30 con 0x7f07e806a270 2023-11-08T08:21:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.699+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07d801a020 con 0x7f07e806a270 2023-11-08T08:21:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.699+0000 7f07e4ff9700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07c80682f0 0x7f07c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.699+0000 7f07e77fe700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07c80682f0 0x7f07c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.699+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f07d8085840 con 0x7f07e806a270 2023-11-08T08:21:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.699+0000 7f07e77fe700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07c80682f0 0x7f07c806a7a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f07d0006fd0 tx=0x7f07d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.703+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07d8052040 con 0x7f07e806a270 2023-11-08T08:21:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.882+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07e80743f0 con 0x7f07c80682f0 2023-11-08T08:21:36.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.890+0000 7f07e4ff9700 1 -- 172.21.15.116:0/3505538012 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311650 (secure 0 0 0) 0x7f07e80743f0 con 0x7f07c80682f0 2023-11-08T08:21:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07c80682f0 msgr2=0x7f07c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07c80682f0 0x7f07c806a7a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f07d0006fd0 tx=0x7f07d0008040 comp rx=0 tx=0).stop 2023-11-08T08:21:36.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 msgr2=0x7f07e80af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:36.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 0x7f07e80af290 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c520 tx=0x7f07d800c8e0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 shutdown_connections 2023-11-08T08:21:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f07e815cc80 0x7f07e81f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e806a270 0x7f07e80af290 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f07c80682f0 0x7f07c806a7a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 --2- 172.21.15.116:0/3505538012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e806e570 0x7f07e81f5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.893+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 >> 172.21.15.116:0/3505538012 conn(0x7f07e806c010 msgr2=0x7f07e80ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:36.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.894+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 shutdown_connections 2023-11-08T08:21:36.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:36.894+0000 7f07ee98f700 1 -- 172.21.15.116:0/3505538012 wait complete. 2023-11-08T08:21:36.900 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:21:36.906 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2023-11-08T08:21:36.391323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":77636,"kb_used_data":6468,"kb_used_omap":0,"kb_used_meta":71104,"kb_avail":754897084,"statfs":{"total":773094113280,"available":773014614016,"internally_reserved":0,"allocated":6623232,"data_stored":3525647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72810496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.709177"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099377+0000","last_change":"2023-11-08T08:21:32.107547+0000","last_active":"2023-11-08T08:21:33.099377+0000","last_peered":"2023-11-08T08:21:33.099377+0000","last_clean":"2023-11-08T08:21:33.099377+0000","last_became_active":"2023-11-08T08:21:32.107232+0000","last_became_peered":"2023-11-08T08:21:32.107232+0000","last_unstale":"2023-11-08T08:21:33.099377+0000","last_undegraded":"2023-11-08T08:21:33.099377+0000","last_fullsized":"2023-11-08T08:21:33.099377+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:58:51.265070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291215+0000","last_change":"2023-11-08T08:21:32.100334+0000","last_active":"2023-11-08T08:21:33.291215+0000","last_peered":"2023-11-08T08:21:33.291215+0000","last_clean":"2023-11-08T08:21:33.291215+0000","last_became_active":"2023-11-08T08:21:32.100061+0000","last_became_peered":"2023-11-08T08:21:32.100061+0000","last_unstale":"2023-11-08T08:21:33.291215+0000","last_undegraded":"2023-11-08T08:21:33.291215+0000","last_fullsized":"2023-11-08T08:21:33.291215+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:40:41.657914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290752+0000","last_change":"2023-11-08T08:21:32.103981+0000","last_active":"2023-11-08T08:21:33.290752+0000","last_peered":"2023-11-08T08:21:33.290752+0000","last_clean":"2023-11-08T08:21:33.290752+0000","last_became_active":"2023-11-08T08:21:32.103757+0000","last_became_peered":"2023-11-08T08:21:32.103757+0000","last_unstale":"2023-11-08T08:21:33.290752+0000","last_undegraded":"2023-11-08T08:21:33.290752+0000","last_fullsized":"2023-11-08T08:21:33.290752+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:57:30.530999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291052+0000","last_change":"2023-11-08T08:21:32.106439+0000","last_active":"2023-11-08T08:21:33.291052+0000","last_peered":"2023-11-08T08:21:33.291052+0000","last_clean":"2023-11-08T08:21:33.291052+0000","last_became_active":"2023-11-08T08:21:32.103271+0000","last_became_peered":"2023-11-08T08:21:32.103271+0000","last_unstale":"2023-11-08T08:21:33.291052+0000","last_undegraded":"2023-11-08T08:21:33.291052+0000","last_fullsized":"2023-11-08T08:21:33.291052+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:43:31.204987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290696+0000","last_change":"2023-11-08T08:21:32.102308+0000","last_active":"2023-11-08T08:21:33.290696+0000","last_peered":"2023-11-08T08:21:33.290696+0000","last_clean":"2023-11-08T08:21:33.290696+0000","last_became_active":"2023-11-08T08:21:32.102029+0000","last_became_peered":"2023-11-08T08:21:32.102029+0000","last_unstale":"2023-11-08T08:21:33.290696+0000","last_undegraded":"2023-11-08T08:21:33.290696+0000","last_fullsized":"2023-11-08T08:21:33.290696+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:44.888334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290708+0000","last_change":"2023-11-08T08:21:32.107208+0000","last_active":"2023-11-08T08:21:33.290708+0000","last_peered":"2023-11-08T08:21:33.290708+0000","last_clean":"2023-11-08T08:21:33.290708+0000","last_became_active":"2023-11-08T08:21:32.106514+0000","last_became_peered":"2023-11-08T08:21:32.106514+0000","last_unstale":"2023-11-08T08:21:33.290708+0000","last_undegraded":"2023-11-08T08:21:33.290708+0000","last_fullsized":"2023-11-08T08:21:33.290708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:26.497082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098938+0000","last_change":"2023-11-08T08:21:32.111545+0000","last_active":"2023-11-08T08:21:33.098938+0000","last_peered":"2023-11-08T08:21:33.098938+0000","last_clean":"2023-11-08T08:21:33.098938+0000","last_became_active":"2023-11-08T08:21:32.111179+0000","last_became_peered":"2023-11-08T08:21:32.111179+0000","last_unstale":"2023-11-08T08:21:33.098938+0000","last_undegraded":"2023-11-08T08:21:33.098938+0000","last_fullsized":"2023-11-08T08:21:33.098938+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:25:59.550762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290266+0000","last_change":"2023-11-08T08:21:32.099428+0000","last_active":"2023-11-08T08:21:33.290266+0000","last_peered":"2023-11-08T08:21:33.290266+0000","last_clean":"2023-11-08T08:21:33.290266+0000","last_became_active":"2023-11-08T08:21:32.098595+0000","last_became_peered":"2023-11-08T08:21:32.098595+0000","last_unstale":"2023-11-08T08:21:33.290266+0000","last_undegraded":"2023-11-08T08:21:33.290266+0000","last_fullsized":"2023-11-08T08:21:33.290266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:31.579603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291320+0000","last_change":"2023-11-08T08:21:32.108391+0000","last_active":"2023-11-08T08:21:33.291320+0000","last_peered":"2023-11-08T08:21:33.291320+0000","last_clean":"2023-11-08T08:21:33.291320+0000","last_became_active":"2023-11-08T08:21:32.107821+0000","last_became_peered":"2023-11-08T08:21:32.107821+0000","last_unstale":"2023-11-08T08:21:33.291320+0000","last_undegraded":"2023-11-08T08:21:33.291320+0000","last_fullsized":"2023-11-08T08:21:33.291320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:58:56.180020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098609+0000","last_change":"2023-11-08T08:21:32.105640+0000","last_active":"2023-11-08T08:21:33.098609+0000","last_peered":"2023-11-08T08:21:33.098609+0000","last_clean":"2023-11-08T08:21:33.098609+0000","last_became_active":"2023-11-08T08:21:32.105386+0000","last_became_peered":"2023-11-08T08:21:32.105386+0000","last_unstale":"2023-11-08T08:21:33.098609+0000","last_undegraded":"2023-11-08T08:21:33.098609+0000","last_fullsized":"2023-11-08T08:21:33.098609+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:39:47.423926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291075+0000","last_change":"2023-11-08T08:21:32.110541+0000","last_active":"2023-11-08T08:21:33.291075+0000","last_peered":"2023-11-08T08:21:33.291075+0000","last_clean":"2023-11-08T08:21:33.291075+0000","last_became_active":"2023-11-08T08:21:32.109913+0000","last_became_peered":"2023-11-08T08:21:32.109913+0000","last_unstale":"2023-11-08T08:21:33.291075+0000","last_undegraded":"2023-11-08T08:21:33.291075+0000","last_fullsized":"2023-11-08T08:21:33.291075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:44:47.641703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290963+0000","last_change":"2023-11-08T08:21:32.110027+0000","last_active":"2023-11-08T08:21:33.290963+0000","last_peered":"2023-11-08T08:21:33.290963+0000","last_clean":"2023-11-08T08:21:33.290963+0000","last_became_active":"2023-11-08T08:21:32.109655+0000","last_became_peered":"2023-11-08T08:21:32.109655+0000","last_unstale":"2023-11-08T08:21:33.290963+0000","last_undegraded":"2023-11-08T08:21:33.290963+0000","last_fullsized":"2023-11-08T08:21:33.290963+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:49:49.217341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096364+0000","last_change":"2023-11-08T08:21:32.101094+0000","last_active":"2023-11-08T08:21:33.096364+0000","last_peered":"2023-11-08T08:21:33.096364+0000","last_clean":"2023-11-08T08:21:33.096364+0000","last_became_active":"2023-11-08T08:21:32.100844+0000","last_became_peered":"2023-11-08T08:21:32.100844+0000","last_unstale":"2023-11-08T08:21:33.096364+0000","last_undegraded":"2023-11-08T08:21:33.096364+0000","last_fullsized":"2023-11-08T08:21:33.096364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:00:03.701169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096614+0000","last_change":"2023-11-08T08:21:32.100673+0000","last_active":"2023-11-08T08:21:33.096614+0000","last_peered":"2023-11-08T08:21:33.096614+0000","last_clean":"2023-11-08T08:21:33.096614+0000","last_became_active":"2023-11-08T08:21:32.100410+0000","last_became_peered":"2023-11-08T08:21:32.100410+0000","last_unstale":"2023-11-08T08:21:33.096614+0000","last_undegraded":"2023-11-08T08:21:33.096614+0000","last_fullsized":"2023-11-08T08:21:33.096614+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:16.672240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098327+0000","last_change":"2023-11-08T08:21:32.111428+0000","last_active":"2023-11-08T08:21:33.098327+0000","last_peered":"2023-11-08T08:21:33.098327+0000","last_clean":"2023-11-08T08:21:33.098327+0000","last_became_active":"2023-11-08T08:21:32.111166+0000","last_became_peered":"2023-11-08T08:21:32.111166+0000","last_unstale":"2023-11-08T08:21:33.098327+0000","last_undegraded":"2023-11-08T08:21:33.098327+0000","last_fullsized":"2023-11-08T08:21:33.098327+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:43:31.204987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290838+0000","last_change":"2023-11-08T08:21:32.097392+0000","last_active":"2023-11-08T08:21:33.290838+0000","last_peered":"2023-11-08T08:21:33.290838+0000","last_clean":"2023-11-08T08:21:33.290838+0000","last_became_active":"2023-11-08T08:21:32.097119+0000","last_became_peered":"2023-11-08T08:21:32.097119+0000","last_unstale":"2023-11-08T08:21:33.290838+0000","last_undegraded":"2023-11-08T08:21:33.290838+0000","last_fullsized":"2023-11-08T08:21:33.290838+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:16.672240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176453+0000","last_change":"2023-11-08T08:21:32.106868+0000","last_active":"2023-11-08T08:21:33.176453+0000","last_peered":"2023-11-08T08:21:33.176453+0000","last_clean":"2023-11-08T08:21:33.176453+0000","last_became_active":"2023-11-08T08:21:32.106601+0000","last_became_peered":"2023-11-08T08:21:32.106601+0000","last_unstale":"2023-11-08T08:21:33.176453+0000","last_undegraded":"2023-11-08T08:21:33.176453+0000","last_fullsized":"2023-11-08T08:21:33.176453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:21.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291349+0000","last_change":"2023-11-08T08:21:32.109424+0000","last_active":"2023-11-08T08:21:33.291349+0000","last_peered":"2023-11-08T08:21:33.291349+0000","last_clean":"2023-11-08T08:21:33.291349+0000","last_became_active":"2023-11-08T08:21:32.109178+0000","last_became_peered":"2023-11-08T08:21:32.109178+0000","last_unstale":"2023-11-08T08:21:33.291349+0000","last_undegraded":"2023-11-08T08:21:33.291349+0000","last_fullsized":"2023-11-08T08:21:33.291349+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:44:15.097568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291171+0000","last_change":"2023-11-08T08:21:32.110653+0000","last_active":"2023-11-08T08:21:33.291171+0000","last_peered":"2023-11-08T08:21:33.291171+0000","last_clean":"2023-11-08T08:21:33.291171+0000","last_became_active":"2023-11-08T08:21:32.110387+0000","last_became_peered":"2023-11-08T08:21:32.110387+0000","last_unstale":"2023-11-08T08:21:33.291171+0000","last_undegraded":"2023-11-08T08:21:33.291171+0000","last_fullsized":"2023-11-08T08:21:33.291171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:47:17.820009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176288+0000","last_change":"2023-11-08T08:21:32.104375+0000","last_active":"2023-11-08T08:21:33.176288+0000","last_peered":"2023-11-08T08:21:33.176288+0000","last_clean":"2023-11-08T08:21:33.176288+0000","last_became_active":"2023-11-08T08:21:32.104132+0000","last_became_peered":"2023-11-08T08:21:32.104132+0000","last_unstale":"2023-11-08T08:21:33.176288+0000","last_undegraded":"2023-11-08T08:21:33.176288+0000","last_fullsized":"2023-11-08T08:21:33.176288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:58:44.888334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096390+0000","last_change":"2023-11-08T08:21:32.099032+0000","last_active":"2023-11-08T08:21:33.096390+0000","last_peered":"2023-11-08T08:21:33.096390+0000","last_clean":"2023-11-08T08:21:33.096390+0000","last_became_active":"2023-11-08T08:21:32.098175+0000","last_became_peered":"2023-11-08T08:21:32.098175+0000","last_unstale":"2023-11-08T08:21:33.096390+0000","last_undegraded":"2023-11-08T08:21:33.096390+0000","last_fullsized":"2023-11-08T08:21:33.096390+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:28:33.365580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290585+0000","last_change":"2023-11-08T08:21:32.107442+0000","last_active":"2023-11-08T08:21:33.290585+0000","last_peered":"2023-11-08T08:21:33.290585+0000","last_clean":"2023-11-08T08:21:33.290585+0000","last_became_active":"2023-11-08T08:21:32.106950+0000","last_became_peered":"2023-11-08T08:21:32.106950+0000","last_unstale":"2023-11-08T08:21:33.290585+0000","last_undegraded":"2023-11-08T08:21:33.290585+0000","last_fullsized":"2023-11-08T08:21:33.290585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:58:56.180020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290913+0000","last_change":"2023-11-08T08:21:32.102073+0000","last_active":"2023-11-08T08:21:33.290913+0000","last_peered":"2023-11-08T08:21:33.290913+0000","last_clean":"2023-11-08T08:21:33.290913+0000","last_became_active":"2023-11-08T08:21:32.101799+0000","last_became_peered":"2023-11-08T08:21:32.101799+0000","last_unstale":"2023-11-08T08:21:33.290913+0000","last_undegraded":"2023-11-08T08:21:33.290913+0000","last_fullsized":"2023-11-08T08:21:33.290913+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:51.912547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290340+0000","last_change":"2023-11-08T08:21:32.100654+0000","last_active":"2023-11-08T08:21:33.290340+0000","last_peered":"2023-11-08T08:21:33.290340+0000","last_clean":"2023-11-08T08:21:33.290340+0000","last_became_active":"2023-11-08T08:21:32.100408+0000","last_became_peered":"2023-11-08T08:21:32.100408+0000","last_unstale":"2023-11-08T08:21:33.290340+0000","last_undegraded":"2023-11-08T08:21:33.290340+0000","last_fullsized":"2023-11-08T08:21:33.290340+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:32:16.748207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096568+0000","last_change":"2023-11-08T08:21:32.103128+0000","last_active":"2023-11-08T08:21:33.096568+0000","last_peered":"2023-11-08T08:21:33.096568+0000","last_clean":"2023-11-08T08:21:33.096568+0000","last_became_active":"2023-11-08T08:21:32.102847+0000","last_became_peered":"2023-11-08T08:21:32.102847+0000","last_unstale":"2023-11-08T08:21:33.096568+0000","last_undegraded":"2023-11-08T08:21:33.096568+0000","last_fullsized":"2023-11-08T08:21:33.096568+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:11:14.299828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290509+0000","last_change":"2023-11-08T08:21:32.104907+0000","last_active":"2023-11-08T08:21:33.290509+0000","last_peered":"2023-11-08T08:21:33.290509+0000","last_clean":"2023-11-08T08:21:33.290509+0000","last_became_active":"2023-11-08T08:21:32.104654+0000","last_became_peered":"2023-11-08T08:21:32.104654+0000","last_unstale":"2023-11-08T08:21:33.290509+0000","last_undegraded":"2023-11-08T08:21:33.290509+0000","last_fullsized":"2023-11-08T08:21:33.290509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:26:25.345947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096644+0000","last_change":"2023-11-08T08:21:32.103196+0000","last_active":"2023-11-08T08:21:33.096644+0000","last_peered":"2023-11-08T08:21:33.096644+0000","last_clean":"2023-11-08T08:21:33.096644+0000","last_became_active":"2023-11-08T08:21:32.102942+0000","last_became_peered":"2023-11-08T08:21:32.102942+0000","last_unstale":"2023-11-08T08:21:33.096644+0000","last_undegraded":"2023-11-08T08:21:33.096644+0000","last_fullsized":"2023-11-08T08:21:33.096644+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:51:10.113988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099380+0000","last_change":"2023-11-08T08:21:32.109134+0000","last_active":"2023-11-08T08:21:33.099380+0000","last_peered":"2023-11-08T08:21:33.099380+0000","last_clean":"2023-11-08T08:21:33.099380+0000","last_became_active":"2023-11-08T08:21:32.108810+0000","last_became_peered":"2023-11-08T08:21:32.108810+0000","last_unstale":"2023-11-08T08:21:33.099380+0000","last_undegraded":"2023-11-08T08:21:33.099380+0000","last_fullsized":"2023-11-08T08:21:33.099380+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:25:17.999948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291177+0000","last_change":"2023-11-08T08:21:32.107682+0000","last_active":"2023-11-08T08:21:33.291177+0000","last_peered":"2023-11-08T08:21:33.291177+0000","last_clean":"2023-11-08T08:21:33.291177+0000","last_became_active":"2023-11-08T08:21:32.107419+0000","last_became_peered":"2023-11-08T08:21:32.107419+0000","last_unstale":"2023-11-08T08:21:33.291177+0000","last_undegraded":"2023-11-08T08:21:33.291177+0000","last_fullsized":"2023-11-08T08:21:33.291177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:44:44.990866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291185+0000","last_change":"2023-11-08T08:21:32.107532+0000","last_active":"2023-11-08T08:21:33.291185+0000","last_peered":"2023-11-08T08:21:33.291185+0000","last_clean":"2023-11-08T08:21:33.291185+0000","last_became_active":"2023-11-08T08:21:32.106993+0000","last_became_peered":"2023-11-08T08:21:32.106993+0000","last_unstale":"2023-11-08T08:21:33.291185+0000","last_undegraded":"2023-11-08T08:21:33.291185+0000","last_fullsized":"2023-11-08T08:21:33.291185+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:40:41.657914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290580+0000","last_change":"2023-11-08T08:21:32.101225+0000","last_active":"2023-11-08T08:21:33.290580+0000","last_peered":"2023-11-08T08:21:33.290580+0000","last_clean":"2023-11-08T08:21:33.290580+0000","last_became_active":"2023-11-08T08:21:32.100444+0000","last_became_peered":"2023-11-08T08:21:32.100444+0000","last_unstale":"2023-11-08T08:21:33.290580+0000","last_undegraded":"2023-11-08T08:21:33.290580+0000","last_fullsized":"2023-11-08T08:21:33.290580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:15:59.123082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099029+0000","last_change":"2023-11-08T08:21:32.108546+0000","last_active":"2023-11-08T08:21:33.099029+0000","last_peered":"2023-11-08T08:21:33.099029+0000","last_clean":"2023-11-08T08:21:33.099029+0000","last_became_active":"2023-11-08T08:21:32.108262+0000","last_became_peered":"2023-11-08T08:21:32.108262+0000","last_unstale":"2023-11-08T08:21:33.099029+0000","last_undegraded":"2023-11-08T08:21:33.099029+0000","last_fullsized":"2023-11-08T08:21:33.099029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:32.238411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291040+0000","last_change":"2023-11-08T08:21:32.109010+0000","last_active":"2023-11-08T08:21:33.291040+0000","last_peered":"2023-11-08T08:21:33.291040+0000","last_clean":"2023-11-08T08:21:33.291040+0000","last_became_active":"2023-11-08T08:21:32.107037+0000","last_became_peered":"2023-11-08T08:21:32.107037+0000","last_unstale":"2023-11-08T08:21:33.291040+0000","last_undegraded":"2023-11-08T08:21:33.291040+0000","last_fullsized":"2023-11-08T08:21:33.291040+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:53:35.233536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1f","version":"55'4","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099434+0000","last_change":"2023-11-08T08:21:31.704889+0000","last_active":"2023-11-08T08:21:33.099434+0000","last_peered":"2023-11-08T08:21:33.099434+0000","last_clean":"2023-11-08T08:21:33.099434+0000","last_became_active":"2023-11-08T08:21:31.704617+0000","last_became_peered":"2023-11-08T08:21:31.704617+0000","last_unstale":"2023-11-08T08:21:33.099434+0000","last_undegraded":"2023-11-08T08:21:33.099434+0000","last_fullsized":"2023-11-08T08:21:33.099434+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:50.904536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290462+0000","last_change":"2023-11-08T08:21:32.102306+0000","last_active":"2023-11-08T08:21:33.290462+0000","last_peered":"2023-11-08T08:21:33.290462+0000","last_clean":"2023-11-08T08:21:33.290462+0000","last_became_active":"2023-11-08T08:21:32.102048+0000","last_became_peered":"2023-11-08T08:21:32.102048+0000","last_unstale":"2023-11-08T08:21:33.290462+0000","last_undegraded":"2023-11-08T08:21:33.290462+0000","last_fullsized":"2023-11-08T08:21:33.290462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:27:07.315328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1e","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099297+0000","last_change":"2023-11-08T08:21:31.706003+0000","last_active":"2023-11-08T08:21:33.099297+0000","last_peered":"2023-11-08T08:21:33.099297+0000","last_clean":"2023-11-08T08:21:33.099297+0000","last_became_active":"2023-11-08T08:21:31.705747+0000","last_became_peered":"2023-11-08T08:21:31.705747+0000","last_unstale":"2023-11-08T08:21:33.099297+0000","last_undegraded":"2023-11-08T08:21:33.099297+0000","last_fullsized":"2023-11-08T08:21:33.099297+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:25:14.455165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291244+0000","last_change":"2023-11-08T08:21:32.100306+0000","last_active":"2023-11-08T08:21:33.291244+0000","last_peered":"2023-11-08T08:21:33.291244+0000","last_clean":"2023-11-08T08:21:33.291244+0000","last_became_active":"2023-11-08T08:21:32.099999+0000","last_became_peered":"2023-11-08T08:21:32.099999+0000","last_unstale":"2023-11-08T08:21:33.291244+0000","last_undegraded":"2023-11-08T08:21:33.291244+0000","last_fullsized":"2023-11-08T08:21:33.291244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:44:10.321033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099583+0000","last_change":"2023-11-08T08:21:31.703084+0000","last_active":"2023-11-08T08:21:33.099583+0000","last_peered":"2023-11-08T08:21:33.099583+0000","last_clean":"2023-11-08T08:21:33.099583+0000","last_became_active":"2023-11-08T08:21:31.702806+0000","last_became_peered":"2023-11-08T08:21:31.702806+0000","last_unstale":"2023-11-08T08:21:33.099583+0000","last_undegraded":"2023-11-08T08:21:33.099583+0000","last_fullsized":"2023-11-08T08:21:33.099583+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:53:59.457113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291441+0000","last_change":"2023-11-08T08:21:32.098934+0000","last_active":"2023-11-08T08:21:33.291441+0000","last_peered":"2023-11-08T08:21:33.291441+0000","last_clean":"2023-11-08T08:21:33.291441+0000","last_became_active":"2023-11-08T08:21:32.098662+0000","last_became_peered":"2023-11-08T08:21:32.098662+0000","last_unstale":"2023-11-08T08:21:33.291441+0000","last_undegraded":"2023-11-08T08:21:33.291441+0000","last_fullsized":"2023-11-08T08:21:33.291441+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:26:04.016526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1c","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290897+0000","last_change":"2023-11-08T08:21:31.702201+0000","last_active":"2023-11-08T08:21:33.290897+0000","last_peered":"2023-11-08T08:21:33.290897+0000","last_clean":"2023-11-08T08:21:33.290897+0000","last_became_active":"2023-11-08T08:21:31.701923+0000","last_became_peered":"2023-11-08T08:21:31.701923+0000","last_unstale":"2023-11-08T08:21:33.290897+0000","last_undegraded":"2023-11-08T08:21:33.290897+0000","last_fullsized":"2023-11-08T08:21:33.290897+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:28:32.357569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.175803+0000","last_change":"2023-11-08T08:21:32.104074+0000","last_active":"2023-11-08T08:21:33.175803+0000","last_peered":"2023-11-08T08:21:33.175803+0000","last_clean":"2023-11-08T08:21:33.175803+0000","last_became_active":"2023-11-08T08:21:32.103795+0000","last_became_peered":"2023-11-08T08:21:32.103795+0000","last_unstale":"2023-11-08T08:21:33.175803+0000","last_undegraded":"2023-11-08T08:21:33.175803+0000","last_fullsized":"2023-11-08T08:21:33.175803+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:38:20.555180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176213+0000","last_change":"2023-11-08T08:21:31.696909+0000","last_active":"2023-11-08T08:21:33.176213+0000","last_peered":"2023-11-08T08:21:33.176213+0000","last_clean":"2023-11-08T08:21:33.176213+0000","last_became_active":"2023-11-08T08:21:31.696589+0000","last_became_peered":"2023-11-08T08:21:31.696589+0000","last_unstale":"2023-11-08T08:21:33.176213+0000","last_undegraded":"2023-11-08T08:21:33.176213+0000","last_fullsized":"2023-11-08T08:21:33.176213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:32:15.740195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098929+0000","last_change":"2023-11-08T08:21:32.105999+0000","last_active":"2023-11-08T08:21:33.098929+0000","last_peered":"2023-11-08T08:21:33.098929+0000","last_clean":"2023-11-08T08:21:33.098929+0000","last_became_active":"2023-11-08T08:21:32.102873+0000","last_became_peered":"2023-11-08T08:21:32.102873+0000","last_unstale":"2023-11-08T08:21:33.098929+0000","last_undegraded":"2023-11-08T08:21:33.098929+0000","last_fullsized":"2023-11-08T08:21:33.098929+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:28:54.922372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1a","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290808+0000","last_change":"2023-11-08T08:21:31.703524+0000","last_active":"2023-11-08T08:21:33.290808+0000","last_peered":"2023-11-08T08:21:33.290808+0000","last_clean":"2023-11-08T08:21:33.290808+0000","last_became_active":"2023-11-08T08:21:31.703256+0000","last_became_peered":"2023-11-08T08:21:31.703256+0000","last_unstale":"2023-11-08T08:21:33.290808+0000","last_undegraded":"2023-11-08T08:21:33.290808+0000","last_fullsized":"2023-11-08T08:21:33.290808+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:59:18.212331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098415+0000","last_change":"2023-11-08T08:21:32.101609+0000","last_active":"2023-11-08T08:21:33.098415+0000","last_peered":"2023-11-08T08:21:33.098415+0000","last_clean":"2023-11-08T08:21:33.098415+0000","last_became_active":"2023-11-08T08:21:32.101327+0000","last_became_peered":"2023-11-08T08:21:32.101327+0000","last_unstale":"2023-11-08T08:21:33.098415+0000","last_undegraded":"2023-11-08T08:21:33.098415+0000","last_fullsized":"2023-11-08T08:21:33.098415+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:19:17.115635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099238+0000","last_change":"2023-11-08T08:21:31.707009+0000","last_active":"2023-11-08T08:21:33.099238+0000","last_peered":"2023-11-08T08:21:33.099238+0000","last_clean":"2023-11-08T08:21:33.099238+0000","last_became_active":"2023-11-08T08:21:31.706743+0000","last_became_peered":"2023-11-08T08:21:31.706743+0000","last_unstale":"2023-11-08T08:21:33.099238+0000","last_undegraded":"2023-11-08T08:21:33.099238+0000","last_fullsized":"2023-11-08T08:21:33.099238+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:58.572397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.175967+0000","last_change":"2023-11-08T08:21:32.104046+0000","last_active":"2023-11-08T08:21:33.175967+0000","last_peered":"2023-11-08T08:21:33.175967+0000","last_clean":"2023-11-08T08:21:33.175967+0000","last_became_active":"2023-11-08T08:21:32.100653+0000","last_became_peered":"2023-11-08T08:21:32.100653+0000","last_unstale":"2023-11-08T08:21:33.175967+0000","last_undegraded":"2023-11-08T08:21:33.175967+0000","last_fullsized":"2023-11-08T08:21:33.175967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:26:25.345947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290824+0000","last_change":"2023-11-08T08:21:31.703174+0000","last_active":"2023-11-08T08:21:33.290824+0000","last_peered":"2023-11-08T08:21:33.290824+0000","last_clean":"2023-11-08T08:21:33.290824+0000","last_became_active":"2023-11-08T08:21:31.702884+0000","last_became_peered":"2023-11-08T08:21:31.702884+0000","last_unstale":"2023-11-08T08:21:33.290824+0000","last_undegraded":"2023-11-08T08:21:33.290824+0000","last_fullsized":"2023-11-08T08:21:33.290824+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:14:00.278020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099494+0000","last_change":"2023-11-08T08:21:32.105793+0000","last_active":"2023-11-08T08:21:33.099494+0000","last_peered":"2023-11-08T08:21:33.099494+0000","last_clean":"2023-11-08T08:21:33.099494+0000","last_became_active":"2023-11-08T08:21:32.105456+0000","last_became_peered":"2023-11-08T08:21:32.105456+0000","last_unstale":"2023-11-08T08:21:33.099494+0000","last_undegraded":"2023-11-08T08:21:33.099494+0000","last_fullsized":"2023-11-08T08:21:33.099494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:56:19.112133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291392+0000","last_change":"2023-11-08T08:21:31.707361+0000","last_active":"2023-11-08T08:21:33.291392+0000","last_peered":"2023-11-08T08:21:33.291392+0000","last_clean":"2023-11-08T08:21:33.291392+0000","last_became_active":"2023-11-08T08:21:31.707052+0000","last_became_peered":"2023-11-08T08:21:31.707052+0000","last_unstale":"2023-11-08T08:21:33.291392+0000","last_undegraded":"2023-11-08T08:21:33.291392+0000","last_fullsized":"2023-11-08T08:21:33.291392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:50.904536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176171+0000","last_change":"2023-11-08T08:21:32.106778+0000","last_active":"2023-11-08T08:21:33.176171+0000","last_peered":"2023-11-08T08:21:33.176171+0000","last_clean":"2023-11-08T08:21:33.176171+0000","last_became_active":"2023-11-08T08:21:32.106444+0000","last_became_peered":"2023-11-08T08:21:32.106444+0000","last_unstale":"2023-11-08T08:21:33.176171+0000","last_undegraded":"2023-11-08T08:21:33.176171+0000","last_fullsized":"2023-11-08T08:21:33.176171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:38:37.762000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096765+0000","last_change":"2023-11-08T08:21:31.704456+0000","last_active":"2023-11-08T08:21:33.096765+0000","last_peered":"2023-11-08T08:21:33.096765+0000","last_clean":"2023-11-08T08:21:33.096765+0000","last_became_active":"2023-11-08T08:21:31.704190+0000","last_became_peered":"2023-11-08T08:21:31.704190+0000","last_unstale":"2023-11-08T08:21:33.096765+0000","last_undegraded":"2023-11-08T08:21:33.096765+0000","last_fullsized":"2023-11-08T08:21:33.096765+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:36:54.332089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096597+0000","last_change":"2023-11-08T08:21:32.100162+0000","last_active":"2023-11-08T08:21:33.096597+0000","last_peered":"2023-11-08T08:21:33.096597+0000","last_clean":"2023-11-08T08:21:33.096597+0000","last_became_active":"2023-11-08T08:21:32.099890+0000","last_became_peered":"2023-11-08T08:21:32.099890+0000","last_unstale":"2023-11-08T08:21:33.096597+0000","last_undegraded":"2023-11-08T08:21:33.096597+0000","last_fullsized":"2023-11-08T08:21:33.096597+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:36:16.410656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.15","version":"55'3","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290557+0000","last_change":"2023-11-08T08:21:31.709724+0000","last_active":"2023-11-08T08:21:33.290557+0000","last_peered":"2023-11-08T08:21:33.290557+0000","last_clean":"2023-11-08T08:21:33.290557+0000","last_became_active":"2023-11-08T08:21:31.709428+0000","last_became_peered":"2023-11-08T08:21:31.709428+0000","last_unstale":"2023-11-08T08:21:33.290557+0000","last_undegraded":"2023-11-08T08:21:33.290557+0000","last_fullsized":"2023-11-08T08:21:33.290557+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:26:03.008514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291171+0000","last_change":"2023-11-08T08:21:32.101170+0000","last_active":"2023-11-08T08:21:33.291171+0000","last_peered":"2023-11-08T08:21:33.291171+0000","last_clean":"2023-11-08T08:21:33.291171+0000","last_became_active":"2023-11-08T08:21:32.100904+0000","last_became_peered":"2023-11-08T08:21:32.100904+0000","last_unstale":"2023-11-08T08:21:33.291171+0000","last_undegraded":"2023-11-08T08:21:33.291171+0000","last_fullsized":"2023-11-08T08:21:33.291171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:58:07.081698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"55'6","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290998+0000","last_change":"2023-11-08T08:21:31.700460+0000","last_active":"2023-11-08T08:21:33.290998+0000","last_peered":"2023-11-08T08:21:33.290998+0000","last_clean":"2023-11-08T08:21:33.290998+0000","last_became_active":"2023-11-08T08:21:31.700153+0000","last_became_peered":"2023-11-08T08:21:31.700153+0000","last_unstale":"2023-11-08T08:21:33.290998+0000","last_undegraded":"2023-11-08T08:21:33.290998+0000","last_fullsized":"2023-11-08T08:21:33.290998+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:04:51.874801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098231+0000","last_change":"2023-11-08T08:21:32.106213+0000","last_active":"2023-11-08T08:21:33.098231+0000","last_peered":"2023-11-08T08:21:33.098231+0000","last_clean":"2023-11-08T08:21:33.098231+0000","last_became_active":"2023-11-08T08:21:32.105962+0000","last_became_peered":"2023-11-08T08:21:32.105962+0000","last_unstale":"2023-11-08T08:21:33.098231+0000","last_undegraded":"2023-11-08T08:21:33.098231+0000","last_fullsized":"2023-11-08T08:21:33.098231+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:44:47.641703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291230+0000","last_change":"2023-11-08T08:21:31.704944+0000","last_active":"2023-11-08T08:21:33.291230+0000","last_peered":"2023-11-08T08:21:33.291230+0000","last_clean":"2023-11-08T08:21:33.291230+0000","last_became_active":"2023-11-08T08:21:31.703990+0000","last_became_peered":"2023-11-08T08:21:31.703990+0000","last_unstale":"2023-11-08T08:21:33.291230+0000","last_undegraded":"2023-11-08T08:21:33.291230+0000","last_fullsized":"2023-11-08T08:21:33.291230+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:24:33.396545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098351+0000","last_change":"2023-11-08T08:21:32.105400+0000","last_active":"2023-11-08T08:21:33.098351+0000","last_peered":"2023-11-08T08:21:33.098351+0000","last_clean":"2023-11-08T08:21:33.098351+0000","last_became_active":"2023-11-08T08:21:32.105143+0000","last_became_peered":"2023-11-08T08:21:32.105143+0000","last_unstale":"2023-11-08T08:21:33.098351+0000","last_undegraded":"2023-11-08T08:21:33.098351+0000","last_fullsized":"2023-11-08T08:21:33.098351+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:49:49.217341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098437+0000","last_change":"2023-11-08T08:21:31.699946+0000","last_active":"2023-11-08T08:21:33.098437+0000","last_peered":"2023-11-08T08:21:33.098437+0000","last_clean":"2023-11-08T08:21:33.098437+0000","last_became_active":"2023-11-08T08:21:31.699463+0000","last_became_peered":"2023-11-08T08:21:31.699463+0000","last_unstale":"2023-11-08T08:21:33.098437+0000","last_undegraded":"2023-11-08T08:21:33.098437+0000","last_fullsized":"2023-11-08T08:21:33.098437+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:23:31.230400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098526+0000","last_change":"2023-11-08T08:21:32.107991+0000","last_active":"2023-11-08T08:21:33.098526+0000","last_peered":"2023-11-08T08:21:33.098526+0000","last_clean":"2023-11-08T08:21:33.098526+0000","last_became_active":"2023-11-08T08:21:32.107723+0000","last_became_peered":"2023-11-08T08:21:32.107723+0000","last_unstale":"2023-11-08T08:21:33.098526+0000","last_undegraded":"2023-11-08T08:21:33.098526+0000","last_fullsized":"2023-11-08T08:21:33.098526+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:53:35.233536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290387+0000","last_change":"2023-11-08T08:21:31.703510+0000","last_active":"2023-11-08T08:21:33.290387+0000","last_peered":"2023-11-08T08:21:33.290387+0000","last_clean":"2023-11-08T08:21:33.290387+0000","last_became_active":"2023-11-08T08:21:31.703226+0000","last_became_peered":"2023-11-08T08:21:31.703226+0000","last_unstale":"2023-11-08T08:21:33.290387+0000","last_undegraded":"2023-11-08T08:21:33.290387+0000","last_fullsized":"2023-11-08T08:21:33.290387+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T20:19:16.107623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.099017+0000","last_change":"2023-11-08T08:21:32.107787+0000","last_active":"2023-11-08T08:21:33.099017+0000","last_peered":"2023-11-08T08:21:33.099017+0000","last_clean":"2023-11-08T08:21:33.099017+0000","last_became_active":"2023-11-08T08:21:32.107456+0000","last_became_peered":"2023-11-08T08:21:32.107456+0000","last_unstale":"2023-11-08T08:21:33.099017+0000","last_undegraded":"2023-11-08T08:21:33.099017+0000","last_fullsized":"2023-11-08T08:21:33.099017+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:44:10.321033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"55'8","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098250+0000","last_change":"2023-11-08T08:21:31.697573+0000","last_active":"2023-11-08T08:21:33.098250+0000","last_peered":"2023-11-08T08:21:33.098250+0000","last_clean":"2023-11-08T08:21:33.098250+0000","last_became_active":"2023-11-08T08:21:31.697410+0000","last_became_peered":"2023-11-08T08:21:31.697410+0000","last_unstale":"2023-11-08T08:21:33.098250+0000","last_undegraded":"2023-11-08T08:21:33.098250+0000","last_fullsized":"2023-11-08T08:21:33.098250+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:56:18.104122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176210+0000","last_change":"2023-11-08T08:21:32.110171+0000","last_active":"2023-11-08T08:21:33.176210+0000","last_peered":"2023-11-08T08:21:33.176210+0000","last_clean":"2023-11-08T08:21:33.176210+0000","last_became_active":"2023-11-08T08:21:32.109871+0000","last_became_peered":"2023-11-08T08:21:32.109871+0000","last_unstale":"2023-11-08T08:21:33.176210+0000","last_undegraded":"2023-11-08T08:21:33.176210+0000","last_fullsized":"2023-11-08T08:21:33.176210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:58:07.081698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291444+0000","last_change":"2023-11-08T08:21:31.708438+0000","last_active":"2023-11-08T08:21:33.291444+0000","last_peered":"2023-11-08T08:21:33.291444+0000","last_clean":"2023-11-08T08:21:33.291444+0000","last_became_active":"2023-11-08T08:21:31.708206+0000","last_became_peered":"2023-11-08T08:21:31.708206+0000","last_unstale":"2023-11-08T08:21:33.291444+0000","last_undegraded":"2023-11-08T08:21:33.291444+0000","last_fullsized":"2023-11-08T08:21:33.291444+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:51:09.105977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176135+0000","last_change":"2023-11-08T08:21:32.112116+0000","last_active":"2023-11-08T08:21:33.176135+0000","last_peered":"2023-11-08T08:21:33.176135+0000","last_clean":"2023-11-08T08:21:33.176135+0000","last_became_active":"2023-11-08T08:21:32.111476+0000","last_became_peered":"2023-11-08T08:21:32.111476+0000","last_unstale":"2023-11-08T08:21:33.176135+0000","last_undegraded":"2023-11-08T08:21:33.176135+0000","last_fullsized":"2023-11-08T08:21:33.176135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:57:30.530999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291394+0000","last_change":"2023-11-08T08:21:31.705424+0000","last_active":"2023-11-08T08:21:33.291394+0000","last_peered":"2023-11-08T08:21:33.291394+0000","last_clean":"2023-11-08T08:21:33.291394+0000","last_became_active":"2023-11-08T08:21:31.704184+0000","last_became_peered":"2023-11-08T08:21:31.704184+0000","last_unstale":"2023-11-08T08:21:33.291394+0000","last_undegraded":"2023-11-08T08:21:33.291394+0000","last_fullsized":"2023-11-08T08:21:33.291394+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:25:14.455165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290507+0000","last_change":"2023-11-08T08:21:32.104411+0000","last_active":"2023-11-08T08:21:33.290507+0000","last_peered":"2023-11-08T08:21:33.290507+0000","last_clean":"2023-11-08T08:21:33.290507+0000","last_became_active":"2023-11-08T08:21:32.104130+0000","last_became_peered":"2023-11-08T08:21:32.104130+0000","last_unstale":"2023-11-08T08:21:33.290507+0000","last_undegraded":"2023-11-08T08:21:33.290507+0000","last_fullsized":"2023-11-08T08:21:33.290507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:50:43.841421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290924+0000","last_change":"2023-11-08T08:21:31.700555+0000","last_active":"2023-11-08T08:21:33.290924+0000","last_peered":"2023-11-08T08:21:33.290924+0000","last_clean":"2023-11-08T08:21:33.290924+0000","last_became_active":"2023-11-08T08:21:31.699947+0000","last_became_peered":"2023-11-08T08:21:31.699947+0000","last_unstale":"2023-11-08T08:21:33.290924+0000","last_undegraded":"2023-11-08T08:21:33.290924+0000","last_fullsized":"2023-11-08T08:21:33.290924+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:36:15.402645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290777+0000","last_change":"2023-11-08T08:21:32.098843+0000","last_active":"2023-11-08T08:21:33.290777+0000","last_peered":"2023-11-08T08:21:33.290777+0000","last_clean":"2023-11-08T08:21:33.290777+0000","last_became_active":"2023-11-08T08:21:32.098583+0000","last_became_peered":"2023-11-08T08:21:32.098583+0000","last_unstale":"2023-11-08T08:21:33.290777+0000","last_undegraded":"2023-11-08T08:21:33.290777+0000","last_fullsized":"2023-11-08T08:21:33.290777+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:38:37.762000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291098+0000","last_change":"2023-11-08T08:21:31.706776+0000","last_active":"2023-11-08T08:21:33.291098+0000","last_peered":"2023-11-08T08:21:33.291098+0000","last_clean":"2023-11-08T08:21:33.291098+0000","last_became_active":"2023-11-08T08:21:31.706439+0000","last_became_peered":"2023-11-08T08:21:31.706439+0000","last_unstale":"2023-11-08T08:21:33.291098+0000","last_undegraded":"2023-11-08T08:21:33.291098+0000","last_fullsized":"2023-11-08T08:21:33.291098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:30:20.916773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290810+0000","last_change":"2023-11-08T08:21:32.108904+0000","last_active":"2023-11-08T08:21:33.290810+0000","last_peered":"2023-11-08T08:21:33.290810+0000","last_clean":"2023-11-08T08:21:33.290810+0000","last_became_active":"2023-11-08T08:21:32.108655+0000","last_became_peered":"2023-11-08T08:21:32.108655+0000","last_unstale":"2023-11-08T08:21:33.290810+0000","last_undegraded":"2023-11-08T08:21:33.290810+0000","last_fullsized":"2023-11-08T08:21:33.290810+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:50:26.497082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096462+0000","last_change":"2023-11-08T08:21:31.702419+0000","last_active":"2023-11-08T08:21:33.096462+0000","last_peered":"2023-11-08T08:21:33.096462+0000","last_clean":"2023-11-08T08:21:33.096462+0000","last_became_active":"2023-11-08T08:21:31.702072+0000","last_became_peered":"2023-11-08T08:21:31.702072+0000","last_unstale":"2023-11-08T08:21:33.096462+0000","last_undegraded":"2023-11-08T08:21:33.096462+0000","last_fullsized":"2023-11-08T08:21:33.096462+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:09:21.883003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096382+0000","last_change":"2023-11-08T08:21:32.100545+0000","last_active":"2023-11-08T08:21:33.096382+0000","last_peered":"2023-11-08T08:21:33.096382+0000","last_clean":"2023-11-08T08:21:33.096382+0000","last_became_active":"2023-11-08T08:21:32.099949+0000","last_became_peered":"2023-11-08T08:21:32.099949+0000","last_unstale":"2023-11-08T08:21:33.096382+0000","last_undegraded":"2023-11-08T08:21:33.096382+0000","last_fullsized":"2023-11-08T08:21:33.096382+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:59.580409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098969+0000","last_change":"2023-11-08T08:21:31.705077+0000","last_active":"2023-11-08T08:21:33.098969+0000","last_peered":"2023-11-08T08:21:33.098969+0000","last_clean":"2023-11-08T08:21:33.098969+0000","last_became_active":"2023-11-08T08:21:31.704297+0000","last_became_peered":"2023-11-08T08:21:31.704297+0000","last_unstale":"2023-11-08T08:21:33.098969+0000","last_undegraded":"2023-11-08T08:21:33.098969+0000","last_fullsized":"2023-11-08T08:21:33.098969+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:15.664229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290840+0000","last_change":"2023-11-08T08:21:32.102465+0000","last_active":"2023-11-08T08:21:33.290840+0000","last_peered":"2023-11-08T08:21:33.290840+0000","last_clean":"2023-11-08T08:21:33.290840+0000","last_became_active":"2023-11-08T08:21:32.102157+0000","last_became_peered":"2023-11-08T08:21:32.102157+0000","last_unstale":"2023-11-08T08:21:33.290840+0000","last_undegraded":"2023-11-08T08:21:33.290840+0000","last_fullsized":"2023-11-08T08:21:33.290840+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:28:54.922372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290805+0000","last_change":"2023-11-08T08:21:31.703438+0000","last_active":"2023-11-08T08:21:33.290805+0000","last_peered":"2023-11-08T08:21:33.290805+0000","last_clean":"2023-11-08T08:21:33.290805+0000","last_became_active":"2023-11-08T08:21:31.703152+0000","last_became_peered":"2023-11-08T08:21:31.703152+0000","last_unstale":"2023-11-08T08:21:33.290805+0000","last_undegraded":"2023-11-08T08:21:33.290805+0000","last_fullsized":"2023-11-08T08:21:33.290805+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:26:32.249721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176103+0000","last_change":"2023-11-08T08:21:32.100884+0000","last_active":"2023-11-08T08:21:33.176103+0000","last_peered":"2023-11-08T08:21:33.176103+0000","last_clean":"2023-11-08T08:21:33.176103+0000","last_became_active":"2023-11-08T08:21:32.100292+0000","last_became_peered":"2023-11-08T08:21:32.100292+0000","last_unstale":"2023-11-08T08:21:33.176103+0000","last_undegraded":"2023-11-08T08:21:33.176103+0000","last_fullsized":"2023-11-08T08:21:33.176103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T19:35:31.579603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290786+0000","last_change":"2023-11-08T08:21:31.705118+0000","last_active":"2023-11-08T08:21:33.290786+0000","last_peered":"2023-11-08T08:21:33.290786+0000","last_clean":"2023-11-08T08:21:33.290786+0000","last_became_active":"2023-11-08T08:21:31.704853+0000","last_became_peered":"2023-11-08T08:21:31.704853+0000","last_unstale":"2023-11-08T08:21:33.290786+0000","last_undegraded":"2023-11-08T08:21:33.290786+0000","last_fullsized":"2023-11-08T08:21:33.290786+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:00:02.693157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096687+0000","last_change":"2023-11-08T08:21:32.104693+0000","last_active":"2023-11-08T08:21:33.096687+0000","last_peered":"2023-11-08T08:21:33.096687+0000","last_clean":"2023-11-08T08:21:33.096687+0000","last_became_active":"2023-11-08T08:21:32.104445+0000","last_became_peered":"2023-11-08T08:21:32.104445+0000","last_unstale":"2023-11-08T08:21:33.096687+0000","last_undegraded":"2023-11-08T08:21:33.096687+0000","last_fullsized":"2023-11-08T08:21:33.096687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:51.912547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290794+0000","last_change":"2023-11-08T08:21:31.703194+0000","last_active":"2023-11-08T08:21:33.290794+0000","last_peered":"2023-11-08T08:21:33.290794+0000","last_clean":"2023-11-08T08:21:33.290794+0000","last_became_active":"2023-11-08T08:21:31.702802+0000","last_became_peered":"2023-11-08T08:21:31.702802+0000","last_unstale":"2023-11-08T08:21:33.290794+0000","last_undegraded":"2023-11-08T08:21:33.290794+0000","last_fullsized":"2023-11-08T08:21:33.290794+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:43:30.196976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.095809+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:21:32.095809+0000","last_peered":"2023-11-08T08:21:32.095809+0000","last_clean":"2023-11-08T08:21:32.095809+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:32.095809+0000","last_undegraded":"2023-11-08T08:21:32.095809+0000","last_fullsized":"2023-11-08T08:21:32.095809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290686+0000","last_change":"2023-11-08T08:21:32.105692+0000","last_active":"2023-11-08T08:21:33.290686+0000","last_peered":"2023-11-08T08:21:33.290686+0000","last_clean":"2023-11-08T08:21:33.290686+0000","last_became_active":"2023-11-08T08:21:32.105425+0000","last_became_peered":"2023-11-08T08:21:32.105425+0000","last_unstale":"2023-11-08T08:21:33.290686+0000","last_undegraded":"2023-11-08T08:21:33.290686+0000","last_fullsized":"2023-11-08T08:21:33.290686+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:01:21.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291167+0000","last_change":"2023-11-08T08:21:31.697385+0000","last_active":"2023-11-08T08:21:33.291167+0000","last_peered":"2023-11-08T08:21:33.291167+0000","last_clean":"2023-11-08T08:21:33.291167+0000","last_became_active":"2023-11-08T08:21:31.697071+0000","last_became_peered":"2023-11-08T08:21:31.697071+0000","last_unstale":"2023-11-08T08:21:33.291167+0000","last_undegraded":"2023-11-08T08:21:33.291167+0000","last_fullsized":"2023-11-08T08:21:33.291167+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T11:14:00.278020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":42,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.089841+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:21:32.089841+0000","last_peered":"2023-11-08T08:21:32.089841+0000","last_clean":"2023-11-08T08:21:32.089841+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:21:32.089841+0000","last_undegraded":"2023-11-08T08:21:32.089841+0000","last_fullsized":"2023-11-08T08:21:32.089841+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290891+0000","last_change":"2023-11-08T08:21:32.106974+0000","last_active":"2023-11-08T08:21:33.290891+0000","last_peered":"2023-11-08T08:21:33.290891+0000","last_clean":"2023-11-08T08:21:33.290891+0000","last_became_active":"2023-11-08T08:21:32.106728+0000","last_became_peered":"2023-11-08T08:21:32.106728+0000","last_unstale":"2023-11-08T08:21:33.290891+0000","last_undegraded":"2023-11-08T08:21:33.290891+0000","last_fullsized":"2023-11-08T08:21:33.290891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T17:44:15.097568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.b","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291086+0000","last_change":"2023-11-08T08:21:31.704398+0000","last_active":"2023-11-08T08:21:33.291086+0000","last_peered":"2023-11-08T08:21:33.291086+0000","last_clean":"2023-11-08T08:21:33.291086+0000","last_became_active":"2023-11-08T08:21:31.704087+0000","last_became_peered":"2023-11-08T08:21:31.704087+0000","last_unstale":"2023-11-08T08:21:33.291086+0000","last_undegraded":"2023-11-08T08:21:33.291086+0000","last_fullsized":"2023-11-08T08:21:33.291086+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:56:50.904536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.090168+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:21:32.090168+0000","last_peered":"2023-11-08T08:21:32.090168+0000","last_clean":"2023-11-08T08:21:32.090168+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:21:32.090168+0000","last_undegraded":"2023-11-08T08:21:32.090168+0000","last_fullsized":"2023-11-08T08:21:32.090168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":198,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.089992+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:21:32.089992+0000","last_peered":"2023-11-08T08:21:32.089992+0000","last_clean":"2023-11-08T08:21:32.089992+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:21:32.089992+0000","last_undegraded":"2023-11-08T08:21:32.089992+0000","last_fullsized":"2023-11-08T08:21:32.089992+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096243+0000","last_change":"2023-11-08T08:21:32.097008+0000","last_active":"2023-11-08T08:21:33.096243+0000","last_peered":"2023-11-08T08:21:33.096243+0000","last_clean":"2023-11-08T08:21:33.096243+0000","last_became_active":"2023-11-08T08:21:32.096755+0000","last_became_peered":"2023-11-08T08:21:32.096755+0000","last_unstale":"2023-11-08T08:21:33.096243+0000","last_undegraded":"2023-11-08T08:21:33.096243+0000","last_fullsized":"2023-11-08T08:21:33.096243+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:59:19.220343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098522+0000","last_change":"2023-11-08T08:21:31.703727+0000","last_active":"2023-11-08T08:21:33.098522+0000","last_peered":"2023-11-08T08:21:33.098522+0000","last_clean":"2023-11-08T08:21:33.098522+0000","last_became_active":"2023-11-08T08:21:31.703434+0000","last_became_peered":"2023-11-08T08:21:31.703434+0000","last_unstale":"2023-11-08T08:21:33.098522+0000","last_undegraded":"2023-11-08T08:21:33.098522+0000","last_fullsized":"2023-11-08T08:21:33.098522+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:25:16.991936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.097258+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:21:32.097258+0000","last_peered":"2023-11-08T08:21:32.097258+0000","last_clean":"2023-11-08T08:21:32.097258+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:21:32.097258+0000","last_undegraded":"2023-11-08T08:21:32.097258+0000","last_fullsized":"2023-11-08T08:21:32.097258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098443+0000","last_change":"2023-11-08T08:21:32.111444+0000","last_active":"2023-11-08T08:21:33.098443+0000","last_peered":"2023-11-08T08:21:33.098443+0000","last_clean":"2023-11-08T08:21:33.098443+0000","last_became_active":"2023-11-08T08:21:32.111181+0000","last_became_peered":"2023-11-08T08:21:32.111181+0000","last_unstale":"2023-11-08T08:21:33.098443+0000","last_undegraded":"2023-11-08T08:21:33.098443+0000","last_fullsized":"2023-11-08T08:21:33.098443+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:26:04.016526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290515+0000","last_change":"2023-11-08T08:21:31.702934+0000","last_active":"2023-11-08T08:21:33.290515+0000","last_peered":"2023-11-08T08:21:33.290515+0000","last_clean":"2023-11-08T08:21:33.290515+0000","last_became_active":"2023-11-08T08:21:31.702659+0000","last_became_peered":"2023-11-08T08:21:31.702659+0000","last_unstale":"2023-11-08T08:21:33.290515+0000","last_undegraded":"2023-11-08T08:21:33.290515+0000","last_fullsized":"2023-11-08T08:21:33.290515+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:15.664229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.092696+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:21:32.092696+0000","last_peered":"2023-11-08T08:21:32.092696+0000","last_clean":"2023-11-08T08:21:32.092696+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:32.092696+0000","last_undegraded":"2023-11-08T08:21:32.092696+0000","last_fullsized":"2023-11-08T08:21:32.092696+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291068+0000","last_change":"2023-11-08T08:21:32.097112+0000","last_active":"2023-11-08T08:21:33.291068+0000","last_peered":"2023-11-08T08:21:33.291068+0000","last_clean":"2023-11-08T08:21:33.291068+0000","last_became_active":"2023-11-08T08:21:32.096382+0000","last_became_peered":"2023-11-08T08:21:32.096382+0000","last_unstale":"2023-11-08T08:21:33.291068+0000","last_undegraded":"2023-11-08T08:21:33.291068+0000","last_fullsized":"2023-11-08T08:21:33.291068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:11:14.299828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290999+0000","last_change":"2023-11-08T08:21:31.705761+0000","last_active":"2023-11-08T08:21:33.290999+0000","last_peered":"2023-11-08T08:21:33.290999+0000","last_clean":"2023-11-08T08:21:33.290999+0000","last_became_active":"2023-11-08T08:21:31.705490+0000","last_became_peered":"2023-11-08T08:21:31.705490+0000","last_unstale":"2023-11-08T08:21:33.290999+0000","last_undegraded":"2023-11-08T08:21:33.290999+0000","last_fullsized":"2023-11-08T08:21:33.290999+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:51:09.105977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.088808+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:21:32.088808+0000","last_peered":"2023-11-08T08:21:32.088808+0000","last_clean":"2023-11-08T08:21:32.088808+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:21:32.088808+0000","last_undegraded":"2023-11-08T08:21:32.088808+0000","last_fullsized":"2023-11-08T08:21:32.088808+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096504+0000","last_change":"2023-11-08T08:21:32.099734+0000","last_active":"2023-11-08T08:21:33.096504+0000","last_peered":"2023-11-08T08:21:33.096504+0000","last_clean":"2023-11-08T08:21:33.096504+0000","last_became_active":"2023-11-08T08:21:32.099461+0000","last_became_peered":"2023-11-08T08:21:32.099461+0000","last_unstale":"2023-11-08T08:21:33.096504+0000","last_undegraded":"2023-11-08T08:21:33.096504+0000","last_fullsized":"2023-11-08T08:21:33.096504+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T09:04:52.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"55'2","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291264+0000","last_change":"2023-11-08T08:21:31.705926+0000","last_active":"2023-11-08T08:21:33.291264+0000","last_peered":"2023-11-08T08:21:33.291264+0000","last_clean":"2023-11-08T08:21:33.291264+0000","last_became_active":"2023-11-08T08:21:31.705693+0000","last_became_peered":"2023-11-08T08:21:31.705693+0000","last_unstale":"2023-11-08T08:21:33.291264+0000","last_undegraded":"2023-11-08T08:21:33.291264+0000","last_fullsized":"2023-11-08T08:21:33.291264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:58.572397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.089337+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:21:32.089337+0000","last_peered":"2023-11-08T08:21:32.089337+0000","last_clean":"2023-11-08T08:21:32.089337+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:21:32.089337+0000","last_undegraded":"2023-11-08T08:21:32.089337+0000","last_fullsized":"2023-11-08T08:21:32.089337+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291131+0000","last_change":"2023-11-08T08:21:32.109819+0000","last_active":"2023-11-08T08:21:33.291131+0000","last_peered":"2023-11-08T08:21:33.291131+0000","last_clean":"2023-11-08T08:21:33.291131+0000","last_became_active":"2023-11-08T08:21:32.109510+0000","last_became_peered":"2023-11-08T08:21:32.109510+0000","last_unstale":"2023-11-08T08:21:33.291131+0000","last_undegraded":"2023-11-08T08:21:33.291131+0000","last_fullsized":"2023-11-08T08:21:33.291131+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:31.682115+0000","last_clean_scrub_stamp":"2023-11-08T08:21:31.682115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:39:47.423926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"55'1","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291499+0000","last_change":"2023-11-08T08:21:31.707415+0000","last_active":"2023-11-08T08:21:33.291499+0000","last_peered":"2023-11-08T08:21:33.291499+0000","last_clean":"2023-11-08T08:21:33.291499+0000","last_became_active":"2023-11-08T08:21:31.706556+0000","last_became_peered":"2023-11-08T08:21:31.706556+0000","last_unstale":"2023-11-08T08:21:33.291499+0000","last_undegraded":"2023-11-08T08:21:33.291499+0000","last_fullsized":"2023-11-08T08:21:33.291499+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:30.674103+0000","last_clean_scrub_stamp":"2023-11-08T08:21:30.674103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:54:15.664229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":43,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-11-08T08:21:32.092202+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:21:32.092202+0000","last_peered":"2023-11-08T08:21:32.092202+0000","last_clean":"2023-11-08T08:21:32.092202+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:21:32.092202+0000","last_undegraded":"2023-11-08T08:21:32.092202+0000","last_fullsized":"2023-11-08T08:21:32.092202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9240,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362600,"statfs":{"total":96636764160,"available":96627302400,"internally_reserved":0,"allocated":671744,"data_stored":292689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63500000000000001,"5min":1.611,"15min":1.611},"last":0.55600000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.71199999999999997,"15min":0.73799999999999999},"min":{"1min":0.33500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.92500000000000004,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70999999999999996,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.501}]},{"osd":2,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.42099999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.98399999999999999,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.438},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.78400000000000003,"15min":0.81899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.73499999999999999,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.57699999999999996}]},{"osd":3,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.45800000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.58899999999999997,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.59199999999999997,"15min":0.60799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71199999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48599999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.623,"5min":0.872,"15min":0.872},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.63300000000000001,"15min":0.64800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.66600000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59599999999999997}]},{"osd":6,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.65500000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.748,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9384,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362456,"statfs":{"total":96636764160,"available":96627154944,"internally_reserved":0,"allocated":688128,"data_stored":293612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66300000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57199999999999995,"5min":1.115,"15min":1.115},"last":0.82499999999999996}]},{"osd":1,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69499999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.38300000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71699999999999997,"5min":1.089,"15min":1.089},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.64000000000000001,"5min":0.996,"15min":0.996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50600000000000001,"15min":0.502},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.65200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.629}]},{"osd":3,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.39100000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66800000000000004,"5min":1.135,"15min":1.135},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.44400000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69999999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72999999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.86099999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.39100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.84299999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.74399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9372,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362468,"statfs":{"total":96636764160,"available":96627167232,"internally_reserved":0,"allocated":675840,"data_stored":292723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.312,"15min":0.308},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.316,"15min":0.308},"min":{"1min":0.249,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.313}]},{"osd":2,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.216,"15min":0.216},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.312,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33600000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.378},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.37,"15min":0.35599999999999998},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.36099999999999999}]},{"osd":4,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.61599999999999999,"15min":0.56699999999999995},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":9.0380000000000003,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":8.5169999999999995,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.496}]},{"osd":5,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":8.9979999999999993,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.46500000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.70999999999999996,"15min":0.67000000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":8.5860000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.67200000000000004,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.60799999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.67500000000000004,"15min":0.629},"min":{"1min":0.45800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":8.6790000000000003,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.69599999999999995,"15min":0.64800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":9.0850000000000009,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.55300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9304,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362536,"statfs":{"total":96636764160,"available":96627236864,"internally_reserved":0,"allocated":671744,"data_stored":292689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.48199999999999998,"15min":0.504},"min":{"1min":0.22900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.71199999999999997,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.50600000000000001,"15min":0.53000000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.69299999999999995,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.374}]},{"osd":2,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.749,"5min":2.032,"15min":2.032},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.42099999999999999,"15min":0.432},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66000000000000003,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.29399999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67300000000000004,"5min":1.788,"15min":1.788},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.311,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.77400000000000002,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.317}]},{"osd":4,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.379},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.47599999999999998}]},{"osd":5,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92300000000000004,"5min":1.825,"15min":1.825},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.627,"15min":0.625},"min":{"1min":0.42099999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.012,"5min":1.75,"15min":1.75},"last":0.51500000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.41599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.98799999999999999,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.58799999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.022,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9504,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362336,"statfs":{"total":96636764160,"available":96627032064,"internally_reserved":0,"allocated":679936,"data_stored":292745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.39500000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.309,"15min":0.309},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.379,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.27400000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.309,"15min":0.309},"min":{"1min":0.255,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.39900000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.377},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.307,"5min":0.22,"15min":0.22},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.46300000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.41799999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.52900000000000003},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.62},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.43099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.52000000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55000000000000004,"15min":0.54200000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.63700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10532,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361308,"statfs":{"total":96636764160,"available":96625979392,"internally_reserved":0,"allocated":1273856,"data_stored":883217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53200000000000003,"5min":1.319,"15min":1.319},"last":0.315},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.58899999999999997,"5min":1.915,"15min":1.915},"last":0.34699999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.41599999999999998,"15min":0.42199999999999999},"min":{"1min":0.25900000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.88300000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.39300000000000002,"15min":0.39900000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.71199999999999997,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65500000000000003,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.625,"5min":1.278,"15min":1.278},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.66300000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82399999999999995,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71599999999999997,"15min":0.72799999999999998},"min":{"1min":0.438,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.94499999999999995,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.41399999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96999999999999997,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":1.371,"15min":1.371},"last":0.64900000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70699999999999996,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89000000000000001,"5min":1.857,"15min":1.857},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.73999999999999999,"15min":0.753},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98499999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.71099999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67600000000000005,"15min":0.68600000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.67800000000000005},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86099999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94362084,"statfs":{"total":96636764160,"available":96626774016,"internally_reserved":0,"allocated":675840,"data_stored":293255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.42099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72799999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.47499999999999998,"15min":0.46800000000000003},"min":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83499999999999996,"5min":1.073,"15min":1.073},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70799999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.60499999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.57699999999999996},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.69999999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66000000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.625,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69999999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.55600000000000005}]},{"osd":7,"last update":"Wed Nov 8 08:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.63800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69699999999999995,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61699999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10544,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361296,"statfs":{"total":96636764160,"available":96625967104,"internally_reserved":0,"allocated":1286144,"data_stored":884717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.93000000000000005,"15min":0.996},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.92000000000000004,"15min":0.98199999999999998},"min":{"1min":0.45100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70299999999999996,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.86199999999999999,"15min":0.91000000000000003},"min":{"1min":0.48099999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72099999999999997,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.86299999999999999,"15min":0.90800000000000003},"min":{"1min":0.437,"5min":0.378,"15min":0.378},"max":{"1min":0.72199999999999998,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.86799999999999999,"15min":0.92300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69199999999999995,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.54200000000000004}]},{"osd":3,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96599999999999997,"15min":1.034},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77700000000000002,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.753,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.495},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36299999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.433,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.432}]},{"osd":7,"last update":"Wed Nov 8 08:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73799999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:21:36.913 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-08T08:21:36.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-11-08T08:21:37.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.072+0000 7fd537f7f700 1 Processor -- start 2023-11-08T08:21:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd537f7f700 1 -- start start 2023-11-08T08:21:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd537f7f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd53006a980 0x7fd5300ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd537f7f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd530071890 0x7fd530071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd537f7f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53015cc10 con 0x7fd5300cf710 2023-11-08T08:21:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd537f7f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53006c9b0 con 0x7fd53006a980 2023-11-08T08:21:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd537f7f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53006cc90 con 0x7fd530071890 2023-11-08T08:21:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.073+0000 7fd53551a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd53006a980 0x7fd5300ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd53006a980 0x7fd5300ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47824/0 (socket says 172.21.15.116:47824) 2023-11-08T08:21:37.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 -- 172.21.15.116:0/713851913 learned_addr learned my addr 172.21.15.116:0/713851913 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:37.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53651c700 1 --2- 172.21.15.116:0/713851913 >> v2:172.21.15.129:3300/0 conn(0x7fd530071890 0x7fd530071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 -- 172.21.15.116:0/713851913 >> v2:172.21.15.129:3300/0 conn(0x7fd530071890 msgr2=0x7fd530071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:37.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 --2- 172.21.15.116:0/713851913 >> v2:172.21.15.129:3300/0 conn(0x7fd530071890 0x7fd530071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 -- 172.21.15.116:0/713851913 >> v1:172.21.15.129:6789/0 conn(0x7fd5300cf710 legacy=0x7fd53006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 -- 172.21.15.116:0/713851913 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53006cf70 con 0x7fd53006a980 2023-11-08T08:21:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd53551a700 1 --2- 172.21.15.116:0/713851913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd53006a980 0x7fd5300ce740 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd53006ad50 tx=0x7fd52c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d476ecae45836f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd534d19700 1 -- 172.21.15.116:0/713851913 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c004230 con 0x7fd53006a980 2023-11-08T08:21:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd534d19700 1 -- 172.21.15.116:0/713851913 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c004390 con 0x7fd53006a980 2023-11-08T08:21:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.074+0000 7fd534d19700 1 -- 172.21.15.116:0/713851913 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0036a0 con 0x7fd53006a980 2023-11-08T08:21:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 -- 172.21.15.116:0/713851913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd53006a980 msgr2=0x7fd5300ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 --2- 172.21.15.116:0/713851913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd53006a980 0x7fd5300ce740 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd53006ad50 tx=0x7fd52c009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 -- 172.21.15.116:0/713851913 shutdown_connections 2023-11-08T08:21:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 --2- 172.21.15.116:0/713851913 >> v2:172.21.15.129:3300/0 conn(0x7fd530071890 0x7fd530071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 --2- 172.21.15.116:0/713851913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd53006a980 0x7fd5300ce740 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 -- 172.21.15.116:0/713851913 >> 172.21.15.116:0/713851913 conn(0x7fd53006c010 msgr2=0x7fd5300cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 -- 172.21.15.116:0/713851913 shutdown_connections 2023-11-08T08:21:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.075+0000 7fd537f7f700 1 -- 172.21.15.116:0/713851913 wait complete. 2023-11-08T08:21:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.076+0000 7fd537f7f700 1 Processor -- start 2023-11-08T08:21:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.076+0000 7fd537f7f700 1 -- start start 2023-11-08T08:21:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd53006a980 0x7fd53015f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd530071890 0x7fd53014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 0x7fd5300b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53015f420 con 0x7fd5300cf710 2023-11-08T08:21:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53014e500 con 0x7fd530071890 2023-11-08T08:21:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53015ce40 con 0x7fd53006a980 2023-11-08T08:21:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 0x7fd5300b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 0x7fd5300b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47828/0 (socket says 172.21.15.116:47828) 2023-11-08T08:21:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 -- 172.21.15.116:0/2314203393 learned_addr learned my addr 172.21.15.116:0/2314203393 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd535d1b700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd53006a980 0x7fd53015f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53551a700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd530071890 0x7fd53014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 -- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd53006a980 msgr2=0x7fd53015f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd53006a980 0x7fd53015f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 -- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd530071890 msgr2=0x7fd53014dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd530071890 0x7fd53014dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 -- 172.21.15.116:0/2314203393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c009550 con 0x7fd5300cf710 2023-11-08T08:21:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd53651c700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 0x7fd5300b0040 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd528007e40 tx=0x7fd52800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52800ed30 con 0x7fd5300cf710 2023-11-08T08:21:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd528014420 con 0x7fd5300cf710 2023-11-08T08:21:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53006d530 con 0x7fd5300cf710 2023-11-08T08:21:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5280147d0 con 0x7fd5300cf710 2023-11-08T08:21:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.077+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53006d250 con 0x7fd5300cf710 2023-11-08T08:21:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.078+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd530160480 con 0x7fd5300cf710 2023-11-08T08:21:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.079+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd528012050 con 0x7fd5300cf710 2023-11-08T08:21:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.079+0000 7fd5267fc700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5140682f0 0x7fd51406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.079+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd528086210 con 0x7fd5300cf710 2023-11-08T08:21:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.079+0000 7fd535d1b700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5140682f0 0x7fd51406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.079+0000 7fd535d1b700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5140682f0 0x7fd51406a7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd520009960 tx=0x7fd5200090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.083+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd528052a70 con 0x7fd5300cf710 2023-11-08T08:21:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.336+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fd5300743f0 con 0x7fd5300cf710 2023-11-08T08:21:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.340+0000 7fd5267fc700 1 -- 172.21.15.116:0/2314203393 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd528056090 con 0x7fd5300cf710 2023-11-08T08:21:37.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5140682f0 msgr2=0x7fd51406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5140682f0 0x7fd51406a7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd520009960 tx=0x7fd5200090e0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 msgr2=0x7fd5300b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 0x7fd5300b0040 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd528007e40 tx=0x7fd52800e3c0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 shutdown_connections 2023-11-08T08:21:37.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5300cf710 0x7fd5300b0040 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd53006a980 0x7fd53015f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd5140682f0 0x7fd51406a7a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 --2- 172.21.15.116:0/2314203393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd530071890 0x7fd53014dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 >> 172.21.15.116:0/2314203393 conn(0x7fd53006c010 msgr2=0x7fd53006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:37.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 shutdown_connections 2023-11-08T08:21:37.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.342+0000 7fd537f7f700 1 -- 172.21.15.116:0/2314203393 wait complete. 2023-11-08T08:21:37.347 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.0 2023-11-08T08:21:37.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:37.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.475+0000 7f1acd657700 1 Processor -- start 2023-11-08T08:21:37.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.475+0000 7f1acd657700 1 -- start start 2023-11-08T08:21:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1acd657700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ac80cd5a0 0x7f1ac80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1acd657700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80b22a0 0x7f1ac80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1acd657700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac80d1ee0 con 0x7f1ac806ab10 2023-11-08T08:21:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1acd657700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac80cdb10 con 0x7f1ac80cd5a0 2023-11-08T08:21:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1acd657700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80d1890 con 0x7f1ac80b22a0 2023-11-08T08:21:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ac80cd5a0 0x7f1ac80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ac80cd5a0 0x7f1ac80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47834/0 (socket says 172.21.15.116:47834) 2023-11-08T08:21:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1271103481 learned_addr learned my addr 172.21.15.116:0/1271103481 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac77fe700 1 --2- 172.21.15.116:0/1271103481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80b22a0 0x7f1ac80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1271103481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80b22a0 msgr2=0x7f1ac80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 --2- 172.21.15.116:0/1271103481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80b22a0 0x7f1ac80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1271103481 >> v1:172.21.15.129:6789/0 conn(0x7f1ac806ab10 legacy=0x7f1ac806aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:37.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1271103481 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac80d1b70 con 0x7f1ac80cd5a0 2023-11-08T08:21:37.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac67fc700 1 --2- 172.21.15.116:0/1271103481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf440 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80098f0 tx=0x7f1ab8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e186be4f27dc7f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:37.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.476+0000 7f1ac5ffb700 1 -- 172.21.15.116:0/1271103481 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8004230 con 0x7f1ac80cd5a0 2023-11-08T08:21:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1ac5ffb700 1 -- 172.21.15.116:0/1271103481 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab8004390 con 0x7f1ac80cd5a0 2023-11-08T08:21:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1ac5ffb700 1 -- 172.21.15.116:0/1271103481 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80036a0 con 0x7f1ac80cd5a0 2023-11-08T08:21:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 -- 172.21.15.116:0/1271103481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80cd5a0 msgr2=0x7f1ac80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 --2- 172.21.15.116:0/1271103481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf440 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80098f0 tx=0x7f1ab8009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 -- 172.21.15.116:0/1271103481 shutdown_connections 2023-11-08T08:21:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 --2- 172.21.15.116:0/1271103481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80b22a0 0x7f1ac80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 --2- 172.21.15.116:0/1271103481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf440 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 -- 172.21.15.116:0/1271103481 >> 172.21.15.116:0/1271103481 conn(0x7f1ac806c520 msgr2=0x7f1ac806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 -- 172.21.15.116:0/1271103481 shutdown_connections 2023-11-08T08:21:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.477+0000 7f1acd657700 1 -- 172.21.15.116:0/1271103481 wait complete. 2023-11-08T08:21:37.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.478+0000 7f1acd657700 1 Processor -- start 2023-11-08T08:21:37.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.478+0000 7f1acd657700 1 -- start start 2023-11-08T08:21:37.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1acd657700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ac806ab10 0x7f1ac815a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1acd657700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 0x7f1ac80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1acd657700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1acd657700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac81595d0 con 0x7f1ac80b22a0 2023-11-08T08:21:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1acd657700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80ccaa0 con 0x7f1ac806ab10 2023-11-08T08:21:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1acd657700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac8158950 con 0x7f1ac80cd5a0 2023-11-08T08:21:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 0x7f1ac80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 0x7f1ac80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47842/0 (socket says 172.21.15.116:47842) 2023-11-08T08:21:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1178169611 learned_addr learned my addr 172.21.15.116:0/1178169611 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac6ffd700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ac806ab10 0x7f1ac815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac77fe700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80cd5a0 msgr2=0x7f1ac80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.479+0000 7f1ac67fc700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ac806ab10 msgr2=0x7f1ac815a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1ac67fc700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ac806ab10 0x7f1ac815a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1ac67fc700 1 -- 172.21.15.116:0/1178169611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8009550 con 0x7f1ac80b22a0 2023-11-08T08:21:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1ac6ffd700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ac806ab10 0x7f1ac815a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1ac67fc700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 0x7f1ac80cf440 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009840 tx=0x7f1ab8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8003e50 con 0x7f1ac80b22a0 2023-11-08T08:21:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab800b660 con 0x7f1ac80b22a0 2023-11-08T08:21:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac80b1ec0 con 0x7f1ac80b22a0 2023-11-08T08:21:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab800ba10 con 0x7f1ac80b22a0 2023-11-08T08:21:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.480+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac815cad0 con 0x7f1ac80b22a0 2023-11-08T08:21:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.481+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac80b2050 con 0x7f1ac80b22a0 2023-11-08T08:21:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.481+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ab8017020 con 0x7f1ac80b22a0 2023-11-08T08:21:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.481+0000 7f1aaf7fe700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ab40682f0 0x7f1ab406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.481+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1ab8087310 con 0x7f1ac80b22a0 2023-11-08T08:21:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.482+0000 7f1ac6ffd700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ab40682f0 0x7f1ab406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:37.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.482+0000 7f1ac6ffd700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ab40682f0 0x7f1ab406a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1abc003fc0 tx=0x7f1abc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:37.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.486+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ab8053b70 con 0x7f1ac80b22a0 2023-11-08T08:21:37.571 INFO:tasks.ceph.mds.d.smithi129.stderr:2023-11-08T08:21:37.570+0000 7f28daf31700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:21:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.709+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ac8159710 con 0x7f1ac80b22a0 2023-11-08T08:21:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.710+0000 7f1aaf7fe700 1 -- 172.21.15.116:0/1178169611 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4661 (secure 0 0 0) 0x7f1ac8159710 con 0x7f1ac80b22a0 2023-11-08T08:21:37.711 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:37.712 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:35.475147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ab40682f0 msgr2=0x7f1ab406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ab40682f0 0x7f1ab406a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1abc003fc0 tx=0x7f1abc009040 comp rx=0 tx=0).stop 2023-11-08T08:21:37.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 msgr2=0x7f1ac80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 0x7f1ac80cf440 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009840 tx=0x7f1ab8004750 comp rx=0 tx=0).stop 2023-11-08T08:21:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 shutdown_connections 2023-11-08T08:21:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1ac80cd5a0 0x7f1ac80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ab40682f0 0x7f1ab406a7a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ac806ab10 0x7f1ac815a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 --2- 172.21.15.116:0/1178169611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac80b22a0 0x7f1ac80cf440 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 >> 172.21.15.116:0/1178169611 conn(0x7f1ac806c520 msgr2=0x7f1ac80b8330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 shutdown_connections 2023-11-08T08:21:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:37.712+0000 7f1acd657700 1 -- 172.21.15.116:0/1178169611 wait complete. 2023-11-08T08:21:37.716 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2023-11-08T08:21:38.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-08T08:21:32.082988+0000', 'modified': '2023-11-08T08:21:35.475147+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6222}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6222': {'gid': 6222, 'name': 'd', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/4191005747', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 4191005747}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 4191005747}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-08T08:21:38.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-08T08:21:38.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-08T08:21:38.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 Processor -- start 2023-11-08T08:21:38.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 -- start start 2023-11-08T08:21:38.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b5c0b0540 0x7f5b5c0c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c070940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0ca7b0 con 0x7f5b5c0ca310 2023-11-08T08:21:38.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5c15a8c0 con 0x7f5b5c0b0540 2023-11-08T08:21:38.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b617d1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0caa10 con 0x7f5b5c15e910 2023-11-08T08:21:38.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.249+0000 7f5b5affd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b5c0b0540 0x7f5b5c0c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b5c0b0540 0x7f5b5c0c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47846/0 (socket says 172.21.15.116:47846) 2023-11-08T08:21:38.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 -- 172.21.15.116:0/2676504205 learned_addr learned my addr 172.21.15.116:0/2676504205 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 -- 172.21.15.116:0/2676504205 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 msgr2=0x7f5b5c070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2676504205 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 -- 172.21.15.116:0/2676504205 >> v1:172.21.15.129:6789/0 conn(0x7f5b5c0ca310 legacy=0x7f5b5c0b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 -- 172.21.15.116:0/2676504205 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0cacf0 con 0x7f5b5c0b0540 2023-11-08T08:21:38.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2676504205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c9bf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009920 tx=0x7f5b50009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ee0c53271f67bb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b59ffb700 1 -- 172.21.15.116:0/2676504205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50004230 con 0x7f5b5c0b0540 2023-11-08T08:21:38.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b59ffb700 1 -- 172.21.15.116:0/2676504205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b50004390 con 0x7f5b5c0b0540 2023-11-08T08:21:38.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.250+0000 7f5b59ffb700 1 -- 172.21.15.116:0/2676504205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50009e70 con 0x7f5b5c0b0540 2023-11-08T08:21:38.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 -- 172.21.15.116:0/2676504205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0b0540 msgr2=0x7f5b5c0c9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2676504205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c9bf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009920 tx=0x7f5b50009c30 comp rx=0 tx=0).stop 2023-11-08T08:21:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 -- 172.21.15.116:0/2676504205 shutdown_connections 2023-11-08T08:21:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2676504205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c9bf0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2676504205 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c070940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 -- 172.21.15.116:0/2676504205 >> 172.21.15.116:0/2676504205 conn(0x7f5b5c0cb540 msgr2=0x7f5b5c0cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 -- 172.21.15.116:0/2676504205 shutdown_connections 2023-11-08T08:21:38.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.251+0000 7f5b617d1700 1 -- 172.21.15.116:0/2676504205 wait complete. 2023-11-08T08:21:38.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.252+0000 7f5b617d1700 1 Processor -- start 2023-11-08T08:21:38.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.252+0000 7f5b617d1700 1 -- start start 2023-11-08T08:21:38.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b617d1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b617d1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 0x7f5b5c0c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b617d1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b617d1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c14f970 con 0x7f5b5c0ca310 2023-11-08T08:21:38.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b617d1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0c5f50 con 0x7f5b5c0b0540 2023-11-08T08:21:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b617d1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0c6230 con 0x7f5b5c15e910 2023-11-08T08:21:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b5a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 0x7f5b5c0c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b5a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 0x7f5b5c0c7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47858/0 (socket says 172.21.15.116:47858) 2023-11-08T08:21:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b5a7fc700 1 -- 172.21.15.116:0/2379674736 learned_addr learned my addr 172.21.15.116:0/2379674736 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.253+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5a7fc700 1 -- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 msgr2=0x7f5b5c14ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c14ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5a7fc700 1 -- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5c0b0540 msgr2=0x7f5b5c0c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5a7fc700 1 -- 172.21.15.116:0/2379674736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b50009550 con 0x7f5b5c0ca310 2023-11-08T08:21:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c14ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 0x7f5b5c0c7230 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00b4a0 tx=0x7f5b4c00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b437fe700 1 -- 172.21.15.116:0/2379674736 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c011830 con 0x7f5b5c0ca310 2023-11-08T08:21:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b437fe700 1 -- 172.21.15.116:0/2379674736 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c011990 con 0x7f5b5c0ca310 2023-11-08T08:21:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c0c93f0 con 0x7f5b5c0ca310 2023-11-08T08:21:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b437fe700 1 -- 172.21.15.116:0/2379674736 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c011d40 con 0x7f5b5c0ca310 2023-11-08T08:21:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.254+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c0c9110 con 0x7f5b5c0ca310 2023-11-08T08:21:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.257+0000 7f5b437fe700 1 -- 172.21.15.116:0/2379674736 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b4c00f050 con 0x7f5b5c0ca310 2023-11-08T08:21:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.257+0000 7f5b437fe700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b480682f0 0x7f5b4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.257+0000 7f5b437fe700 1 -- 172.21.15.116:0/2379674736 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5b4c0857d0 con 0x7f5b5c0ca310 2023-11-08T08:21:38.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.257+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b480682f0 0x7f5b4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.257+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b480682f0 0x7f5b4806a7a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009130 tx=0x7f5b50008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.258+0000 7f5b6d970b80 1 Processor -- start 2023-11-08T08:21:38.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.258+0000 7f5b6d970b80 1 -- start start 2023-11-08T08:21:38.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00ff5f10 0x564d00faf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00fb0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d00f2dde0 con 0x564d00fd8620 2023-11-08T08:21:38.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d00e9a140 con 0x564d00feca80 2023-11-08T08:21:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d00f08db0 con 0x564d00ff5f10 2023-11-08T08:21:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00fb0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00fb0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47860/0 (socket says 172.21.15.116:47860) 2023-11-08T08:21:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/1389620667 learned_addr learned my addr 172.21.15.116:0/1389620667 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5affd700 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00ff5f10 0x564d00faf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00ff5f10 msgr2=0x564d00faf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00ff5f10 0x564d00faf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 msgr2=0x564d00ff62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/1389620667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d00cf6790 con 0x564d00fd8620 2023-11-08T08:21:38.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:38.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00fb0b20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c480 tx=0x7f5b5400c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c61b6b6f4c03dba server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b40ff9700 1 -- 172.21.15.116:0/1389620667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54011070 con 0x564d00fd8620 2023-11-08T08:21:38.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b40ff9700 1 -- 172.21.15.116:0/1389620667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5400ca70 con 0x564d00fd8620 2023-11-08T08:21:38.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.259+0000 7f5b40ff9700 1 -- 172.21.15.116:0/1389620667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400ce20 con 0x564d00fd8620 2023-11-08T08:21:38.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1389620667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 msgr2=0x564d00fb0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00fb0b20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c480 tx=0x7f5b5400c4b0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1389620667 shutdown_connections 2023-11-08T08:21:38.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00fb0b20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00ff5f10 0x564d00faf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1389620667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1389620667 >> 172.21.15.116:0/1389620667 conn(0x564d00fa5ce0 msgr2=0x564d00faeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1389620667 shutdown_connections 2023-11-08T08:21:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.260+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1389620667 wait complete. 2023-11-08T08:21:38.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.261+0000 7f5b6d970b80 1 -- start start 2023-11-08T08:21:38.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.261+0000 7f5b6d970b80 1 Processor -- start 2023-11-08T08:21:38.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00ff74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d00ff8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00ff5f10 0x564d00ff91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d00f02120 con 0x564d00fd8620 2023-11-08T08:21:38.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d00cf7390 con 0x564d00ff5f10 2023-11-08T08:21:38.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d00e9a340 con 0x564d00feca80 2023-11-08T08:21:38.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 10 client.0 _ll_register_callbacks cb 0x564d00ea0400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:38.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00ff74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00ff74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47864/0 (socket says 172.21.15.116:47864) 2023-11-08T08:21:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 -- 172.21.15.116:0/1037947877 learned_addr learned my addr 172.21.15.116:0/1037947877 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d00ff8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00ff5f10 0x564d00ff91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 -- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 msgr2=0x564d00ff8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d00ff8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 -- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00ff5f10 msgr2=0x564d00ff91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00ff5f10 0x564d00ff91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 -- 172.21.15.116:0/1037947877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b54009ca0 con 0x564d00fd8620 2023-11-08T08:21:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b5affd700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00ff74b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009c70 tx=0x7f5b5000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb881edd7d27a5ca server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:38.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b39ffb700 1 -- 172.21.15.116:0/1037947877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5002c040 con 0x564d00fd8620 2023-11-08T08:21:38.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b39ffb700 1 -- 172.21.15.116:0/1037947877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5b50032c40 con 0x564d00fd8620 2023-11-08T08:21:38.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d00f06030 con 0x564d00fd8620 2023-11-08T08:21:38.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b39ffb700 1 -- 172.21.15.116:0/1037947877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b50032e40 con 0x564d00fd8620 2023-11-08T08:21:38.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 10 client.6348 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:38.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.262+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x564d00de30a0 con 0x564d00fd8620 2023-11-08T08:21:38.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b39ffb700 1 -- 172.21.15.116:0/1037947877 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564d00f06030 con 0x564d00fd8620 2023-11-08T08:21:38.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b39ffb700 1 -- 172.21.15.116:0/1037947877 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f5b50003bb0 con 0x564d00fd8620 2023-11-08T08:21:38.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b6d970b80 10 client.6348 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:38.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b6d970b80 10 client.6348 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-08T08:21:38.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b6d970b80 20 client.6348 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:38.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d00fee100 0x564d00fee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b6d970b80 4 client.6348 mds_command: new command op to 6222 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:21:38.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d00d51ef0 con 0x564d00fee100 2023-11-08T08:21:38.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d00fee100 0x564d00fee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b39ffb700 10 client.6348 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.263+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d00fee100 0x564d00fee4d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be9740fa7e02b52 server_cookie=641b68d376923691 in_seq=0 out_seq=0 2023-11-08T08:21:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b39ffb700 1 -- 172.21.15.116:0/1037947877 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564d00d51ef0 con 0x564d00fee100 2023-11-08T08:21:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b39ffb700 10 client.6348 handle_command_reply: tid=0 2023-11-08T08:21:38.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 client.6348 shutdown 2023-11-08T08:21:38.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 msgr2=0x564d00ff74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00ff74b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009c70 tx=0x7f5b5000e000 comp rx=0 tx=0).stop 2023-11-08T08:21:38.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 shutdown_connections 2023-11-08T08:21:38.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00ff5f10 0x564d00ff91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d00fee100 0x564d00fee4d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00fd8620 0x564d00ff74b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b39ffb700 0 client.6348 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/1037947877 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d00ff8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 >> 172.21.15.116:0/1037947877 conn(0x564d00fa5ce0 msgr2=0x564d00faaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 shutdown_connections 2023-11-08T08:21:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.264+0000 7f5b6d970b80 1 -- 172.21.15.116:0/1037947877 wait complete. 2023-11-08T08:21:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.265+0000 7f5b6d970b80 20 client.6348 trim_cache size 0 max 16384 2023-11-08T08:21:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.281+0000 7f5b6d970b80 1 Processor -- start 2023-11-08T08:21:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.281+0000 7f5b6d970b80 1 -- start start 2023-11-08T08:21:38.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00fd8620 0x564d00fdeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d0100f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d00fd8b20 con 0x564d00ff5f10 2023-11-08T08:21:38.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d00ff6cf0 con 0x564d00fd8620 2023-11-08T08:21:38.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d00fab120 con 0x564d00feca80 2023-11-08T08:21:38.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47880/0 (socket says 172.21.15.116:47880) 2023-11-08T08:21:38.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/572584428 learned_addr learned my addr 172.21.15.116:0/572584428 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5affd700 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00fd8620 0x564d00fdeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 msgr2=0x564d0100f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d0100f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00fd8620 msgr2=0x564d00fdeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.282+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00fd8620 0x564d00fdeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/572584428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d00ec92e0 con 0x564d00ff5f10 2023-11-08T08:21:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2220 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c450 tx=0x7f5b5400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b1573904e8cd29 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b417fa700 1 -- 172.21.15.116:0/572584428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400c2e0 con 0x564d00ff5f10 2023-11-08T08:21:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b417fa700 1 -- 172.21.15.116:0/572584428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54007000 con 0x564d00ff5f10 2023-11-08T08:21:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b417fa700 1 -- 172.21.15.116:0/572584428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400cde0 con 0x564d00ff5f10 2023-11-08T08:21:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 -- 172.21.15.116:0/572584428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 msgr2=0x564d00ff2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2220 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c450 tx=0x7f5b5400e000 comp rx=0 tx=0).stop 2023-11-08T08:21:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 -- 172.21.15.116:0/572584428 shutdown_connections 2023-11-08T08:21:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2220 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00fd8620 0x564d00fdeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/572584428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00feca80 0x564d0100f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 -- 172.21.15.116:0/572584428 >> 172.21.15.116:0/572584428 conn(0x564d00fa5ce0 msgr2=0x564d00fad240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 -- 172.21.15.116:0/572584428 shutdown_connections 2023-11-08T08:21:38.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.283+0000 7f5b6d970b80 1 -- 172.21.15.116:0/572584428 wait complete. 2023-11-08T08:21:38.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.284+0000 7f5b6d970b80 1 -- start start 2023-11-08T08:21:38.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.284+0000 7f5b6d970b80 1 Processor -- start 2023-11-08T08:21:38.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00fd8620 0x564d0103e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0105f7c0 con 0x564d00ff5f10 2023-11-08T08:21:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0103e640 con 0x564d00feca80 2023-11-08T08:21:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d00fd3020 con 0x564d00fd8620 2023-11-08T08:21:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 10 client.0 _ll_register_callbacks cb 0x564d00ea0400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47892/0 (socket says 172.21.15.116:47892) 2023-11-08T08:21:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/2963376289 learned_addr learned my addr 172.21.15.116:0/2963376289 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00fd8620 0x564d0103e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00fd8620 msgr2=0x564d0103e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00fd8620 0x564d0103e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 msgr2=0x564d00ff2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 -- 172.21.15.116:0/2963376289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5401a040 con 0x564d00ff5f10 2023-11-08T08:21:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5a7fc700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00fd8620 0x564d0103e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b5b7fe700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2760 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5b54013000 tx=0x7f5b54004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9153110583e6b4f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b39ffb700 1 -- 172.21.15.116:0/2963376289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400c2e0 con 0x564d00ff5f10 2023-11-08T08:21:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b39ffb700 1 -- 172.21.15.116:0/2963376289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5b54006e40 con 0x564d00ff5f10 2023-11-08T08:21:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b5400ca70 con 0x564d00ff5f10 2023-11-08T08:21:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b39ffb700 1 -- 172.21.15.116:0/2963376289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b54020b50 con 0x564d00ff5f10 2023-11-08T08:21:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 10 client.6354 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:38.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.285+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x564d00d571d0 con 0x564d00ff5f10 2023-11-08T08:21:38.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b39ffb700 1 -- 172.21.15.116:0/2963376289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5400ca70 con 0x564d00ff5f10 2023-11-08T08:21:38.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b39ffb700 1 -- 172.21.15.116:0/2963376289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f5b54020cb0 con 0x564d00ff5f10 2023-11-08T08:21:38.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 10 client.6354 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:38.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 10 client.6354 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-08T08:21:38.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 20 client.6354 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:38.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d0104e440 0x564d00fd5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 4 client.6354 mds_command: new command op to 6222 tid=0[{"prefix": "status"}] 2023-11-08T08:21:38.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564d00d87040 con 0x564d0104e440 2023-11-08T08:21:38.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d0104e440 0x564d00fd5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b39ffb700 10 client.6354 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b5affd700 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d0104e440 0x564d00fd5940 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18d6029cb1337383 server_cookie=249da65741b1431 in_seq=0 out_seq=0 2023-11-08T08:21:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b39ffb700 1 -- 172.21.15.116:0/2963376289 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x564d00d87040 con 0x564d0104e440 2023-11-08T08:21:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b39ffb700 10 client.6354 handle_command_reply: tid=0 2023-11-08T08:21:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.286+0000 7f5b6d970b80 1 client.6354 shutdown 2023-11-08T08:21:38.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 msgr2=0x564d00ff2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2760 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5b54013000 tx=0x7f5b54004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 shutdown_connections 2023-11-08T08:21:38.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x564d0104e440 0x564d00fd5940 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d00ff5f10 0x564d00ff2760 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b39ffb700 0 client.6354 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:38.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x564d00fd8620 0x564d0103e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 --2- 172.21.15.116:0/2963376289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564d00feca80 0x564d00ff2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 >> 172.21.15.116:0/2963376289 conn(0x564d00fa5ce0 msgr2=0x564d00fdea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 shutdown_connections 2023-11-08T08:21:38.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 1 -- 172.21.15.116:0/2963376289 wait complete. 2023-11-08T08:21:38.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b6d970b80 20 client.6354 trim_cache size 0 max 16384 2023-11-08T08:21:38.312 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:38.313 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "2f470591-d661-4b50-b7a5-062dfcf46b1d", 2023-11-08T08:21:38.313 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-08T08:21:38.313 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 6222, 2023-11-08T08:21:38.313 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-08T08:21:38.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-08T08:21:38.314 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-08T08:21:38.314 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 6.2027347400000004, 2023-11-08T08:21:38.314 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 32, 2023-11-08T08:21:38.314 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 56, 2023-11-08T08:21:38.314 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 56, 2023-11-08T08:21:38.315 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 11.654202486999999 2023-11-08T08:21:38.315 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:38.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b480682f0 msgr2=0x7f5b4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b480682f0 0x7f5b4806a7a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009130 tx=0x7f5b50008030 comp rx=0 tx=0).stop 2023-11-08T08:21:38.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 msgr2=0x7f5b5c0c7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 0x7f5b5c0c7230 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00b4a0 tx=0x7f5b4c00b7b0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 shutdown_connections 2023-11-08T08:21:38.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5b5c15e910 0x7f5b5c14ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5b480682f0 0x7f5b4806a7a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5c0b0540 0x7f5b5c0c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 --2- 172.21.15.116:0/2379674736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b5c0ca310 0x7f5b5c0c7230 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 >> 172.21.15.116:0/2379674736 conn(0x7f5b5c0cb540 msgr2=0x7f5b5c0b18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 shutdown_connections 2023-11-08T08:21:38.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.287+0000 7f5b617d1700 1 -- 172.21.15.116:0/2379674736 wait complete. 2023-11-08T08:21:38.319 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-08T08:21:38.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.437+0000 7fc1f0cfe700 1 Processor -- start 2023-11-08T08:21:38.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.437+0000 7fc1f0cfe700 1 -- start start 2023-11-08T08:21:38.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1f0cfe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc1ec06a270 0x7fc1ec06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1f0cfe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1f0cfe700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec06ab80 con 0x7fc1ec15cc80 2023-11-08T08:21:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1f0cfe700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec0b2490 con 0x7fc1ec06a270 2023-11-08T08:21:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1f0cfe700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0b1810 con 0x7fc1ec06e570 2023-11-08T08:21:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1e9d9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc1ec15cc80 0x7fc1ec06de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55376/0 (socket says 172.21.15.116:55376) 2023-11-08T08:21:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.438+0000 7fc1e9d9b700 1 -- 172.21.15.116:0/3252425095 learned_addr learned my addr 172.21.15.116:0/3252425095 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:38.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ead9d700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ea59c700 1 --2- 172.21.15.116:0/3252425095 >> v2:172.21.15.129:3300/0 conn(0x7fc1ec06a270 0x7fc1ec06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1e959a700 1 -- 172.21.15.116:0/3252425095 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153089637 0 0) 0x7fc1ec06ab80 con 0x7fc1ec15cc80 2023-11-08T08:21:38.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1e959a700 1 -- 172.21.15.116:0/3252425095 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d80035b0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1e959a700 1 -- 172.21.15.116:0/3252425095 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119626003 0 0) 0x7fc1d80035b0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ea59c700 1 -- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 msgr2=0x7fc1ec070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:38.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ea59c700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ea59c700 1 -- 172.21.15.116:0/3252425095 >> v1:172.21.15.116:6789/0 conn(0x7fc1ec15cc80 legacy=0x7fc1ec06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ea59c700 1 -- 172.21.15.116:0/3252425095 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec0b1af0 con 0x7fc1ec06a270 2023-11-08T08:21:38.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ead9d700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1ea59c700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec06a640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009840 tx=0x7fc1d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c1467570ec8c932 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1e959a700 1 -- 172.21.15.116:0/3252425095 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc001670 con 0x7fc1ec06a270 2023-11-08T08:21:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1e959a700 1 -- 172.21.15.116:0/3252425095 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4004030 con 0x7fc1ec06a270 2023-11-08T08:21:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.439+0000 7fc1e959a700 1 -- 172.21.15.116:0/3252425095 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d40043a0 con 0x7fc1ec06a270 2023-11-08T08:21:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 msgr2=0x7fc1ec06a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec06a640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009840 tx=0x7fc1d4009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3252425095 shutdown_connections 2023-11-08T08:21:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec06a640 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3252425095 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3252425095 >> 172.21.15.116:0/3252425095 conn(0x7fc1ec06c010 msgr2=0x7fc1ec0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3252425095 shutdown_connections 2023-11-08T08:21:38.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.440+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3252425095 wait complete. 2023-11-08T08:21:38.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.441+0000 7fc1f0cfe700 1 Processor -- start 2023-11-08T08:21:38.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.441+0000 7fc1f0cfe700 1 -- start start 2023-11-08T08:21:38.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1f0cfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1f0cfe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec1e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1f0cfe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 0x7fc1ec1f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1f0cfe700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1f2690 con 0x7fc1ec15cc80 2023-11-08T08:21:38.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1f0cfe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1e73a0 con 0x7fc1ec06a270 2023-11-08T08:21:38.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1f0cfe700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1e7680 con 0x7fc1ec06e570 2023-11-08T08:21:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 0x7fc1ec1f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 0x7fc1ec1f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47908/0 (socket says 172.21.15.116:47908) 2023-11-08T08:21:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 -- 172.21.15.116:0/3341206651 learned_addr learned my addr 172.21.15.116:0/3341206651 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 -- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 msgr2=0x7fc1ec1e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec1e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ea59c700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 -- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 msgr2=0x7fc1ec152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.442+0000 7fc1ead9d700 1 -- 172.21.15.116:0/3341206651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4009a70 con 0x7fc1ec15cc80 2023-11-08T08:21:38.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.443+0000 7fc1ead9d700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 0x7fc1ec1f2da0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400efc0 tx=0x7fc1e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.443+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4009cd0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.443+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e400f040 con 0x7fc1ec15cc80 2023-11-08T08:21:38.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.443+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d40094a0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.443+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e400cd60 con 0x7fc1ec15cc80 2023-11-08T08:21:38.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.443+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec0b20b0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.444+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec0b1dd0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.444+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1e4014480 con 0x7fc1ec15cc80 2023-11-08T08:21:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.444+0000 7fc1e2ffd700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1cc0682f0 0x7fc1cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.445+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc1e40865d0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.445+0000 7fc1ea59c700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1cc0682f0 0x7fc1cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:38.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.445+0000 7fc1ea59c700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1cc0682f0 0x7fc1cc06a7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec1e81c0 tx=0x7fc1d4013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:38.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.449+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1e4052ce0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.673+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc1ec15daa0 con 0x7fc1ec15cc80 2023-11-08T08:21:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.673+0000 7fc1e2ffd700 1 -- 172.21.15.116:0/3341206651 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc1e4056300 con 0x7fc1ec15cc80 2023-11-08T08:21:38.675 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-08T08:21:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1cc0682f0 msgr2=0x7fc1cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1cc0682f0 0x7fc1cc06a7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec1e81c0 tx=0x7fc1d4013040 comp rx=0 tx=0).stop 2023-11-08T08:21:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 msgr2=0x7fc1ec1f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 0x7fc1ec1f2da0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400efc0 tx=0x7fc1e400c3a0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 shutdown_connections 2023-11-08T08:21:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec15cc80 0x7fc1ec1f2da0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc1cc0682f0 0x7fc1cc06a7a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1ec06a270 0x7fc1ec152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 --2- 172.21.15.116:0/3341206651 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc1ec06e570 0x7fc1ec1e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 >> 172.21.15.116:0/3341206651 conn(0x7fc1ec06c010 msgr2=0x7fc1ec0ae2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 shutdown_connections 2023-11-08T08:21:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:38.675+0000 7fc1f0cfe700 1 -- 172.21.15.116:0/3341206651 wait complete. 2023-11-08T08:21:38.688 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-08T08:21:38.688 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-08T08:21:38.688 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-08T08:21:38.689 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-08T08:21:38.689 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:38.689 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-08T08:21:38.690 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-08T08:21:38.690 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-08T08:21:38.690 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:21:38.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:38.691 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-08T08:21:38.710 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-08T08:21:38.710 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-08T08:21:38.710 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-08T08:21:38.711 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.711 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-08T08:21:38.711 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.711 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:21:38.711 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:21:38.712 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-08T08:21:38.712 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:21:38.712 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:21:38.712 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-08T08:21:38.713 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-08T08:21:38.713 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:21:38.713 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-08T08:21:38.713 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:21:38.713 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-08T08:21:38.713 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42119sec preferred_lft 42119sec 2023-11-08T08:21:38.714 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-08T08:21:38.714 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.714 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-08T08:21:38.714 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff 2023-11-08T08:21:38.714 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-08T08:21:38.715 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.715 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::8c0:d4ff:fe3b:7f6b/64 scope link 2023-11-08T08:21:38.715 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.715 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:21:38.715 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:76:a3:22:41:c7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-08T08:21:38.716 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bc76:a3ff:fe22:41c7/64 scope link 2023-11-08T08:21:38.716 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.716 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-08T08:21:38.716 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether be:ab:30:29:e8:99 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-08T08:21:38.716 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::bcab:30ff:fe29:e899/64 scope link 2023-11-08T08:21:38.717 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-08T08:21:38.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:38.717 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-08T08:21:38.775 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-08T08:21:38.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:38.777 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-08T08:21:38.836 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-08T08:21:38.836 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-08T08:21:38.837 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-08T08:21:38.837 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-08T08:21:38.837 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-08T08:21:38.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:38.838 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-08T08:21:38.838 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-08T08:21:38.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-08T08:21:38.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-08T08:21:38.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-08T08:21:38.840 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-08T08:21:38.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:38.941 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:21:38.959 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-08T08:21:38.959 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-08T08:21:38.959 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:21:38.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:38.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-08T08:21:39.025 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-08T08:21:39.026 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:21:39.026 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:39.083 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-08T08:21:39.084 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-08T08:21:39.084 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:39.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:21:39.209 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:21:39.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:21:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:21:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:21:39.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:39.277 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:21:39.335 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-08T08:21:39.336 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/) 2023-11-08T08:21:39.378 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-08T08:21:39.406 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-08T08:21:39.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-08T08:21:39.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-08T08:21:39.465+0000 7f1a6c15d5c0 -1 init, newargv = 0x55bc254726c0 newargc=15 2023-11-08T08:21:39.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[146160]: starting ceph client 2023-11-08T08:21:39.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[146160]: starting fuse 2023-11-08T08:21:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-08T08:21:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-08T08:21:39.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:39.483 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-08T08:21:39.541 INFO:teuthology.orchestra.run.smithi116.stdout:46 2023-11-08T08:21:39.541 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46] 2023-11-08T08:21:39.542 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:21:39.542 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-08T08:21:39.542 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-08T08:21:39.543 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:21:39.543 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-08T08:21:39.544 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:21:39.544 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-08T08:21:39.544 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-08T08:21:39.545 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-08T08:21:39.545 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-08T08:21:39.545 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:21:39.546 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-08T08:21:39.546 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-08T08:21:39.546 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-08T08:21:39.547 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-08T08:21:39.547 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:21:39.548 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-08T08:21:39.548 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-08T08:21:39.548 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-08T08:21:39.549 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-08T08:21:39.549 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-08T08:21:39.549 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-08T08:21:39.550 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-08T08:21:39.550 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-08T08:21:39.550 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:21:39.551 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-08T08:21:39.551 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:21:39.663 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.146160.asok 2023-11-08T08:21:39.845 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.146160.asok 2023-11-08T08:21:39.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:39.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.146160.asok status 2023-11-08T08:21:40.002 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:40.002 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-08T08:21:40.002 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b820560f30d310ab2570761b9a294523225d6998", 2023-11-08T08:21:40.003 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-2698-gb820560f (b820560f30d310ab2570761b9a294523225d6998) reef (dev)", 2023-11-08T08:21:40.003 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-08T08:21:40.003 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-08T08:21:40.003 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-08T08:21:40.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "146160", 2023-11-08T08:21:40.004 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-08T08:21:40.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:40.004 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-08T08:21:40.004 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-08T08:21:40.004 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 6366, 2023-11-08T08:21:40.005 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-08T08:21:40.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-08T08:21:40.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-08T08:21:40.005 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6366 2023-11-08T08:21:40.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:40.006 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:21:40.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:21:40.006 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:21:40.006 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1908084773 2023-11-08T08:21:40.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:40.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:40.007 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-08T08:21:40.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-08T08:21:40.007 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-08T08:21:40.007 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1908084773 2023-11-08T08:21:40.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:40.008 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.6366 192.168.0.1:0/1908084773", 2023-11-08T08:21:40.008 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/1908084773", 2023-11-08T08:21:40.008 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-08T08:21:40.008 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 32, 2023-11-08T08:21:40.009 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 56, 2023-11-08T08:21:40.009 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-08T08:21:40.009 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-08T08:21:40.009 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-08T08:21:40.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:40.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:40.012 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-08T08:21:40.012 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:40.031 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-08T08:21:40.031 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:40.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:40.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:40.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:40.100 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2023-11-08T08:21:40.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:40.164 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2023-11-08T08:21:40.226 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-08T08:21:40.226 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-08T08:21:40.227 DEBUG:teuthology.orchestra.run.smithi116:> import stat 2023-11-08T08:21:40.227 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-08T08:21:40.227 DEBUG:teuthology.orchestra.run.smithi116:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2023-11-08T08:21:40.228 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-08T08:21:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:1099511627777 2023-11-08T08:21:40.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:40.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.681+0000 7f3e0fcd4700 1 Processor -- start 2023-11-08T08:21:40.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.681+0000 7f3e0fcd4700 1 -- start start 2023-11-08T08:21:40.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.681+0000 7f3e0fcd4700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e080d0c60 0x7f3e080b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:40.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0fcd4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e0806e400 0x7f3e080d1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0fcd4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e080d1570 con 0x7f3e08160e70 2023-11-08T08:21:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0fcd4700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0806e900 con 0x7f3e080d0c60 2023-11-08T08:21:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0fcd4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08160390 con 0x7f3e0806e400 2023-11-08T08:21:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0d26f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e0806e400 0x7f3e080d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0da70700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e080d0c60 0x7f3e080b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0e271700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3e08160e70 0x7f3e0806f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:38376/0 (socket says 172.21.15.116:38376) 2023-11-08T08:21:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0e271700 1 -- 172.21.15.116:0/355991452 learned_addr learned my addr 172.21.15.116:0/355991452 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0ca6e700 1 -- 172.21.15.116:0/355991452 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856928429 0 0) 0x7f3e080d1570 con 0x7f3e08160e70 2023-11-08T08:21:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0ca6e700 1 -- 172.21.15.116:0/355991452 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df40035b0 con 0x7f3e08160e70 2023-11-08T08:21:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0d26f700 1 -- 172.21.15.116:0/355991452 >> v2:172.21.15.129:3300/0 conn(0x7f3e080d0c60 msgr2=0x7f3e080b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0d26f700 1 --2- 172.21.15.116:0/355991452 >> v2:172.21.15.129:3300/0 conn(0x7f3e080d0c60 0x7f3e080b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0d26f700 1 -- 172.21.15.116:0/355991452 >> v1:172.21.15.129:6789/0 conn(0x7f3e08160e70 legacy=0x7f3e0806f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0d26f700 1 -- 172.21.15.116:0/355991452 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e08160670 con 0x7f3e0806e400 2023-11-08T08:21:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0da70700 1 --2- 172.21.15.116:0/355991452 >> v2:172.21.15.129:3300/0 conn(0x7f3e080d0c60 0x7f3e080b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.682+0000 7f3e0d26f700 1 --2- 172.21.15.116:0/355991452 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e0806e400 0x7f3e080d1030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009840 tx=0x7f3e04009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e326576447ec172 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0ca6e700 1 -- 172.21.15.116:0/355991452 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e04011830 con 0x7f3e0806e400 2023-11-08T08:21:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0ca6e700 1 -- 172.21.15.116:0/355991452 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e04011990 con 0x7f3e0806e400 2023-11-08T08:21:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0ca6e700 1 -- 172.21.15.116:0/355991452 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e04011d00 con 0x7f3e0806e400 2023-11-08T08:21:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/355991452 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e0806e400 msgr2=0x7f3e080d1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/355991452 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e0806e400 0x7f3e080d1030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009840 tx=0x7f3e04009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/355991452 shutdown_connections 2023-11-08T08:21:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/355991452 >> v2:172.21.15.129:3300/0 conn(0x7f3e080d0c60 0x7f3e080b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/355991452 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e0806e400 0x7f3e080d1030 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/355991452 >> 172.21.15.116:0/355991452 conn(0x7f3e0806a270 msgr2=0x7f3e0806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/355991452 shutdown_connections 2023-11-08T08:21:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.683+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/355991452 wait complete. 2023-11-08T08:21:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.684+0000 7f3e0fcd4700 1 Processor -- start 2023-11-08T08:21:40.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.684+0000 7f3e0fcd4700 1 -- start start 2023-11-08T08:21:40.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0fcd4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 0x7f3e08150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:40.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0fcd4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e080d0c60 0x7f3e080b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:40.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0fcd4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e08160e70 0x7f3e0815f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:40.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0fcd4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080b11d0 con 0x7f3e0806e400 2023-11-08T08:21:40.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0fcd4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080b14b0 con 0x7f3e080d0c60 2023-11-08T08:21:40.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0fcd4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080b1790 con 0x7f3e08160e70 2023-11-08T08:21:40.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 0x7f3e08150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:40.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 0x7f3e08150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47924/0 (socket says 172.21.15.116:47924) 2023-11-08T08:21:40.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 -- 172.21.15.116:0/4221191617 learned_addr learned my addr 172.21.15.116:0/4221191617 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:40.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0d26f700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e080d0c60 0x7f3e080b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:40.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0e271700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e08160e70 0x7f3e0815f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:40.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 -- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e08160e70 msgr2=0x7f3e0815f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:40.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e08160e70 0x7f3e0815f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 -- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e080d0c60 msgr2=0x7f3e080b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:40.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e080d0c60 0x7f3e080b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0da70700 1 -- 172.21.15.116:0/4221191617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0094a0 con 0x7f3e0806e400 2023-11-08T08:21:40.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.685+0000 7f3e0d26f700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e080d0c60 0x7f3e080b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:40.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.686+0000 7f3e0da70700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 0x7f3e08150030 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00ebe0 tx=0x7f3dfc009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:40.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.686+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00c910 con 0x7f3e0806e400 2023-11-08T08:21:40.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.686+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc00ca70 con 0x7f3e0806e400 2023-11-08T08:21:40.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.686+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e040094a0 con 0x7f3e0806e400 2023-11-08T08:21:40.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.686+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00cde0 con 0x7f3e0806e400 2023-11-08T08:21:40.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.686+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080d0740 con 0x7f3e0806e400 2023-11-08T08:21:40.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.687+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e08160950 con 0x7f3e0806e400 2023-11-08T08:21:40.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.687+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3dfc019030 con 0x7f3e0806e400 2023-11-08T08:21:40.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.688+0000 7f3dfa7fc700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3dec0682f0 0x7f3dec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:40.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.688+0000 7f3e0d26f700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3dec0682f0 0x7f3dec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:40.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.688+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3dfc085d20 con 0x7f3e0806e400 2023-11-08T08:21:40.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.688+0000 7f3e0d26f700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3dec0682f0 0x7f3dec06a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3e0806e830 tx=0x7f3e04008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:40.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.692+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3dfc052580 con 0x7f3e0806e400 2023-11-08T08:21:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.913+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e081e7680 con 0x7f3e0806e400 2023-11-08T08:21:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.913+0000 7f3dfa7fc700 1 -- 172.21.15.116:0/4221191617 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4661 (secure 0 0 0) 0x7f3dfc017090 con 0x7f3e0806e400 2023-11-08T08:21:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:40.915 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:35.475147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3dec0682f0 msgr2=0x7f3dec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3dec0682f0 0x7f3dec06a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3e0806e830 tx=0x7f3e04008030 comp rx=0 tx=0).stop 2023-11-08T08:21:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 msgr2=0x7f3e08150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 0x7f3e08150030 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00ebe0 tx=0x7f3dfc009ec0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 shutdown_connections 2023-11-08T08:21:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3e08160e70 0x7f3e0815f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e0806e400 0x7f3e08150030 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f3dec0682f0 0x7f3dec06a7a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 --2- 172.21.15.116:0/4221191617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e080d0c60 0x7f3e080b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 >> 172.21.15.116:0/4221191617 conn(0x7f3e0806a270 msgr2=0x7f3e0806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 shutdown_connections 2023-11-08T08:21:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:40.915+0000 7f3e0fcd4700 1 -- 172.21.15.116:0/4221191617 wait complete. 2023-11-08T08:21:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2023-11-08T08:21:40.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-11-08T08:21:41.094 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2023-11-08T08:21:41.094 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-08T08:21:41.095 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2023-11-08T08:21:41.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 Processor -- start 2023-11-08T08:21:41.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 -- start start 2023-11-08T08:21:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x5598750b6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x559875025e70 0x5598750a4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559874f83000 con 0x55987501fb90 2023-11-08T08:21:41.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x559874f15fe0 con 0x5598750b7630 2023-11-08T08:21:41.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d85e340c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598750abcd0 con 0x559875025e70 2023-11-08T08:21:41.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d7bf20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x5598750b6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d7bf20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x5598750b6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47940/0 (socket says 172.21.15.116:47940) 2023-11-08T08:21:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.139+0000 7f8d7bf20700 1 -- 172.21.15.116:0/2593108453 learned_addr learned my addr 172.21.15.116:0/2593108453 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7c721700 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7b71f700 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x559875025e70 0x5598750a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7bf20700 1 -- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x559875025e70 msgr2=0x5598750a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7bf20700 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x559875025e70 0x5598750a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7bf20700 1 -- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598750b7630 msgr2=0x559875007010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7bf20700 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7bf20700 1 -- 172.21.15.116:0/2593108453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70005d20 con 0x55987501fb90 2023-11-08T08:21:41.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7b71f700 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x559875025e70 0x5598750a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7bf20700 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x5598750b6880 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b390 tx=0x7f8d6c00b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4126b52b23e6a8db server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7a71d700 1 -- 172.21.15.116:0/2593108453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c014070 con 0x55987501fb90 2023-11-08T08:21:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7a71d700 1 -- 172.21.15.116:0/2593108453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c00f3f0 con 0x55987501fb90 2023-11-08T08:21:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d7a71d700 1 -- 172.21.15.116:0/2593108453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c00f7a0 con 0x55987501fb90 2023-11-08T08:21:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 -- 172.21.15.116:0/2593108453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 msgr2=0x5598750b6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x5598750b6880 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b390 tx=0x7f8d6c00b6a0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 -- 172.21.15.116:0/2593108453 shutdown_connections 2023-11-08T08:21:41.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x5598750b6880 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/2593108453 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x559875025e70 0x5598750a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 -- 172.21.15.116:0/2593108453 >> 172.21.15.116:0/2593108453 conn(0x559875026620 msgr2=0x5598750066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 -- 172.21.15.116:0/2593108453 shutdown_connections 2023-11-08T08:21:41.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.140+0000 7f8d85e340c0 1 -- 172.21.15.116:0/2593108453 wait complete. 2023-11-08T08:21:41.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.141+0000 7f8d85e340c0 1 Processor -- start 2023-11-08T08:21:41.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.141+0000 7f8d85e340c0 1 -- start start 2023-11-08T08:21:41.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x559874fdaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559875025e70 0x5598750a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598750b5ce0 con 0x55987501fb90 2023-11-08T08:21:41.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x559874fd9c90 con 0x559875025e70 2023-11-08T08:21:41.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x559874fd9ee0 con 0x5598750b7630 2023-11-08T08:21:41.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x559874fdaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x559874fdaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47946/0 (socket says 172.21.15.116:47946) 2023-11-08T08:21:41.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 -- 172.21.15.116:0/465256930 learned_addr learned my addr 172.21.15.116:0/465256930 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7b71f700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559875025e70 0x5598750a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7c721700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 -- 172.21.15.116:0/465256930 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5598750b7630 msgr2=0x559875007010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5598750b7630 0x559875007010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 -- 172.21.15.116:0/465256930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559875025e70 msgr2=0x5598750a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559875025e70 0x5598750a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 -- 172.21.15.116:0/465256930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c001520 con 0x55987501fb90 2023-11-08T08:21:41.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7bf20700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x559874fdaf80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x55987500c810 tx=0x7f8d6c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7971b700 1 -- 172.21.15.116:0/465256930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c014070 con 0x55987501fb90 2023-11-08T08:21:41.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7971b700 1 -- 172.21.15.116:0/465256930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c001b20 con 0x55987501fb90 2023-11-08T08:21:41.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d7971b700 1 -- 172.21.15.116:0/465256930 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c01b640 con 0x55987501fb90 2023-11-08T08:21:41.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559874f21830 con 0x55987501fb90 2023-11-08T08:21:41.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.142+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559874f2e360 con 0x55987501fb90 2023-11-08T08:21:41.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d7971b700 1 -- 172.21.15.116:0/465256930 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d6c01b7a0 con 0x55987501fb90 2023-11-08T08:21:41.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d7971b700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8d68066c10 0x7f8d680690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d7971b700 1 -- 172.21.15.116:0/465256930 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8d6c00a040 con 0x55987501fb90 2023-11-08T08:21:41.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d7b71f700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8d68066c10 0x7f8d680690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x559875141420 0x55987513eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x55987513efe0 con 0x559875141420 2023-11-08T08:21:41.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d7c721700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x559875141420 0x55987513eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.144+0000 7f8d7b71f700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8d68066c10 0x7f8d680690c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x5598750262b0 tx=0x7f8d7000e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d7c721700 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x559875141420 0x55987513eaa0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d7c721700 1 -- 172.21.15.116:0/465256930 <== osd.3 v2:172.21.15.116:6808/121155 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+94 (crc 0 0 0) 0x7f8d74002600 con 0x559875141420 2023-11-08T08:21:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x559875141420 msgr2=0x55987513eaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x559875141420 0x55987513eaa0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8d68066c10 msgr2=0x7f8d680690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f8d68066c10 0x7f8d680690c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x5598750262b0 tx=0x7f8d7000e000 comp rx=0 tx=0).stop 2023-11-08T08:21:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 msgr2=0x559874fdaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 --2- 172.21.15.116:0/465256930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55987501fb90 0x559874fdaf80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x55987500c810 tx=0x7f8d6c000bc0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d7c721700 1 -- 172.21.15.116:0/465256930 reap_dead start 2023-11-08T08:21:41.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 shutdown_connections 2023-11-08T08:21:41.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 >> 172.21.15.116:0/465256930 conn(0x559875026620 msgr2=0x5598750a23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 shutdown_connections 2023-11-08T08:21:41.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.145+0000 7f8d85e340c0 1 -- 172.21.15.116:0/465256930 wait complete. 2023-11-08T08:21:41.165 INFO:teuthology.orchestra.run.smithi116.stdout:Xfiledir_test 2023-11-08T08:21:41.165 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:41.165 DEBUG:teuthology.orchestra.run.smithi116:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-11-08T08:21:41.221 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1099511627777, 2023-11-08T08:21:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "ancestors": [ 2023-11-08T08:21:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "dirino": 1099511627776, 2023-11-08T08:21:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "dname": "file", 2023-11-08T08:21:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "version": 2 2023-11-08T08:21:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-08T08:21:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "dirino": 1, 2023-11-08T08:21:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "dname": "dir_test", 2023-11-08T08:21:41.224 INFO:teuthology.orchestra.run.smithi116.stdout: "version": 4 2023-11-08T08:21:41.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:41.224 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-11-08T08:21:41.224 INFO:teuthology.orchestra.run.smithi116.stdout: "pool": 10, 2023-11-08T08:21:41.224 INFO:teuthology.orchestra.run.smithi116.stdout: "old_pools": [] 2023-11-08T08:21:41.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:41.225 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:41.226 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2023-11-08T08:21:41.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.311+0000 7f695f7f50c0 1 Processor -- start 2023-11-08T08:21:41.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.311+0000 7f695f7f50c0 1 -- start start 2023-11-08T08:21:41.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f695f7f50c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1d66540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f695f7f50c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 0x55c2b1ce6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f695f7f50c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1d791a0 0x55c2b1ce6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f695f7f50c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b1bdbfe0 con 0x55c2b1cea7e0 2023-11-08T08:21:41.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f695f7f50c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b1ccf9b0 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f695f7f50c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b1ccf6a0 con 0x55c2b1d791a0 2023-11-08T08:21:41.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 0x55c2b1ce6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 0x55c2b1ce6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47950/0 (socket says 172.21.15.116:47950) 2023-11-08T08:21:41.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 -- 172.21.15.116:0/2300164376 learned_addr learned my addr 172.21.15.116:0/2300164376 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69558e1700 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1d66540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69560e2700 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1d791a0 0x55c2b1ce6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 -- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1d791a0 msgr2=0x55c2b1ce6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1d791a0 0x55c2b1ce6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 -- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1ccb7e0 msgr2=0x55c2b1d66540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1d66540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 -- 172.21.15.116:0/2300164376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948005d20 con 0x55c2b1cea7e0 2023-11-08T08:21:41.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69560e2700 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1d791a0 0x55c2b1ce6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f69550e0700 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 0x55c2b1ce6870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f694c009750 tx=0x7f694c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55be2a0d187e2d4e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f6947fff700 1 -- 172.21.15.116:0/2300164376 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c00f040 con 0x55c2b1cea7e0 2023-11-08T08:21:41.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f6947fff700 1 -- 172.21.15.116:0/2300164376 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c008870 con 0x55c2b1cea7e0 2023-11-08T08:21:41.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.312+0000 7f6947fff700 1 -- 172.21.15.116:0/2300164376 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c005e10 con 0x55c2b1cea7e0 2023-11-08T08:21:41.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 -- 172.21.15.116:0/2300164376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 msgr2=0x55c2b1ce6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 0x55c2b1ce6870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f694c009750 tx=0x7f694c005a00 comp rx=0 tx=0).stop 2023-11-08T08:21:41.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 -- 172.21.15.116:0/2300164376 shutdown_connections 2023-11-08T08:21:41.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1d791a0 0x55c2b1ce6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1d66540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/2300164376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1cea7e0 0x55c2b1ce6870 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 -- 172.21.15.116:0/2300164376 >> 172.21.15.116:0/2300164376 conn(0x55c2b1ce9150 msgr2=0x55c2b1ceb890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 -- 172.21.15.116:0/2300164376 shutdown_connections 2023-11-08T08:21:41.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 -- 172.21.15.116:0/2300164376 wait complete. 2023-11-08T08:21:41.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 Processor -- start 2023-11-08T08:21:41.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.313+0000 7f695f7f50c0 1 -- start start 2023-11-08T08:21:41.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f695f7f50c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1cebc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f695f7f50c0 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1cea7e0 0x55c2b1d65760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f695f7f50c0 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1d791a0 0x55c2b1cd1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f695f7f50c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b1cd1470 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f695f7f50c0 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b1d76ec0 con 0x55c2b1d791a0 2023-11-08T08:21:41.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f695f7f50c0 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b1ce5b50 con 0x55c2b1cea7e0 2023-11-08T08:21:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1cebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1cebc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47962/0 (socket says 172.21.15.116:47962) 2023-11-08T08:21:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 -- 172.21.15.116:0/203877212 learned_addr learned my addr 172.21.15.116:0/203877212 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69550e0700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1cea7e0 0x55c2b1d65760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69560e2700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1d791a0 0x55c2b1cd1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 -- 172.21.15.116:0/203877212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1cea7e0 msgr2=0x55c2b1d65760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55c2b1cea7e0 0x55c2b1d65760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 -- 172.21.15.116:0/203877212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1d791a0 msgr2=0x55c2b1cd1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2b1d791a0 0x55c2b1cd1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.314+0000 7f69558e1700 1 -- 172.21.15.116:0/203877212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c009410 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.315+0000 7f69558e1700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1cebc40 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6948011000 tx=0x7f6948002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.315+0000 7f6946ffd700 1 -- 172.21.15.116:0/203877212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948012820 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.315+0000 7f6946ffd700 1 -- 172.21.15.116:0/203877212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948012980 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.315+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c2b1bf4120 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.315+0000 7f6946ffd700 1 -- 172.21.15.116:0/203877212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948012d30 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.315+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2b1be7830 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.316+0000 7f6946ffd700 1 -- 172.21.15.116:0/203877212 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6948026460 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.316+0000 7f6946ffd700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6938066c10 0x7f69380690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.316+0000 7f6946ffd700 1 -- 172.21.15.116:0/203877212 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f694800ae40 con 0x55c2b1ccb7e0 2023-11-08T08:21:41.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.316+0000 7f69550e0700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6938066c10 0x7f69380690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.317+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x55c2b1d77b30 0x55c2b1e0ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.317+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v8 -- 0x55c2b1e0f010 con 0x55c2b1d77b30 2023-11-08T08:21:41.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.317+0000 7f69560e2700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x55c2b1d77b30 0x55c2b1e0ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.317+0000 7f69550e0700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6938066c10 0x7f69380690c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x55c2b1d6fcb0 tx=0x7f694c012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.317+0000 7f69560e2700 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x55c2b1d77b30 0x55c2b1e0ead0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f69560e2700 1 -- 172.21.15.116:0/203877212 <== osd.3 v2:172.21.15.116:6808/121155 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v56'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f6950001930 con 0x55c2b1d77b30 2023-11-08T08:21:41.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x55c2b1d77b30 msgr2=0x55c2b1e0ead0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x55c2b1d77b30 0x55c2b1e0ead0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6938066c10 msgr2=0x7f69380690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6938066c10 0x7f69380690c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x55c2b1d6fcb0 tx=0x7f694c012000 comp rx=0 tx=0).stop 2023-11-08T08:21:41.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1ccb7e0 msgr2=0x55c2b1cebc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 --2- 172.21.15.116:0/203877212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b1ccb7e0 0x55c2b1cebc40 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6948011000 tx=0x7f6948002fb0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f69560e2700 1 -- 172.21.15.116:0/203877212 reap_dead start 2023-11-08T08:21:41.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 shutdown_connections 2023-11-08T08:21:41.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 >> 172.21.15.116:0/203877212 conn(0x55c2b1ce9150 msgr2=0x55c2b1ca0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 shutdown_connections 2023-11-08T08:21:41.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.318+0000 7f695f7f50c0 1 -- 172.21.15.116:0/203877212 wait complete. 2023-11-08T08:21:41.340 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2023-11-08T08:21:41.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.462+0000 7fdadf4e3700 1 Processor -- start 2023-11-08T08:21:41.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.462+0000 7fdadf4e3700 1 -- start start 2023-11-08T08:21:41.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadf4e3700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdad80ca280 0x7fdad815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadf4e3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad80c4c40 con 0x7fdad806e3e0 2023-11-08T08:21:41.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadf4e3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad815bf70 con 0x7fdad80c50d0 2023-11-08T08:21:41.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadf4e3700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad806e7b0 con 0x7fdad80ca280 2023-11-08T08:21:41.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadd27f700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdad806e3e0 0x7fdad806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55394/0 (socket says 172.21.15.116:55394) 2023-11-08T08:21:41.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadd27f700 1 -- 172.21.15.116:0/463455870 learned_addr learned my addr 172.21.15.116:0/463455870 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:41.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434634782 0 0) 0x7fdad80c4c40 con 0x7fdad806e3e0 2023-11-08T08:21:41.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac40035b0 con 0x7fdad806e3e0 2023-11-08T08:21:41.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.463+0000 7fdadca7e700 1 --2- 172.21.15.116:0/463455870 >> v2:172.21.15.129:3300/0 conn(0x7fdad80ca280 0x7fdad815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2614817114 0 0) 0x7fdac40035b0 con 0x7fdad806e3e0 2023-11-08T08:21:41.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 >> v2:172.21.15.129:3300/0 conn(0x7fdad80ca280 msgr2=0x7fdad815ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 --2- 172.21.15.116:0/463455870 >> v2:172.21.15.129:3300/0 conn(0x7fdad80ca280 0x7fdad815ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 >> v1:172.21.15.129:6789/0 conn(0x7fdad80c50d0 legacy=0x7fdad8070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad80ca760 con 0x7fdad806e3e0 2023-11-08T08:21:41.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fdad4001670 con 0x7fdad806e3e0 2023-11-08T08:21:41.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 >> v1:172.21.15.116:6789/0 conn(0x7fdad806e3e0 legacy=0x7fdad806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac4003a00 0x7fdac4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac4004850 0x7fdac4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4005200 0x7fdac400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad806e7b0 con 0x7fdac4003a00 2023-11-08T08:21:41.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac400c080 con 0x7fdac4004850 2023-11-08T08:21:41.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac400c240 con 0x7fdac4005200 2023-11-08T08:21:41.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac4003a00 0x7fdac4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadca7e700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac4004850 0x7fdac4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadda80700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4005200 0x7fdac400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 -- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4005200 msgr2=0x7fdac400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4005200 0x7fdac400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 -- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac4004850 msgr2=0x7fdac4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac4004850 0x7fdac4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 -- 172.21.15.116:0/463455870 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac40035b0 con 0x7fdac4003a00 2023-11-08T08:21:41.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadca7e700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac4004850 0x7fdac4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadda80700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4005200 0x7fdac400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadda80700 1 -- 172.21.15.116:0/463455870 reap_dead start 2023-11-08T08:21:41.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdadd27f700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac4003a00 0x7fdac4003e10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007d30 tx=0x7fdac4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61bd0c6dfe3fd1b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.464+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad400cdb0 con 0x7fdac4003a00 2023-11-08T08:21:41.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad4005960 con 0x7fdac4003a00 2023-11-08T08:21:41.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdacffff700 1 -- 172.21.15.116:0/463455870 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4005d10 con 0x7fdac4003a00 2023-11-08T08:21:41.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 -- 172.21.15.116:0/463455870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac4003a00 msgr2=0x7fdac4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac4003a00 0x7fdac4003e10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007d30 tx=0x7fdac4005880 comp rx=0 tx=0).stop 2023-11-08T08:21:41.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 -- 172.21.15.116:0/463455870 shutdown_connections 2023-11-08T08:21:41.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/463455870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac4003a00 0x7fdac4003e10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 -- 172.21.15.116:0/463455870 >> 172.21.15.116:0/463455870 conn(0x7fdad80cfc30 msgr2=0x7fdad80d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 -- 172.21.15.116:0/463455870 shutdown_connections 2023-11-08T08:21:41.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.465+0000 7fdadf4e3700 1 -- 172.21.15.116:0/463455870 wait complete. 2023-11-08T08:21:41.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.466+0000 7fdadf4e3700 1 Processor -- start 2023-11-08T08:21:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.466+0000 7fdadf4e3700 1 -- start start 2023-11-08T08:21:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadf4e3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4003a00 0x7fdad8142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadf4e3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdad80c7eb0 0x7fdad80c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadf4e3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 0x7fdad80b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadf4e3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad8142e40 con 0x7fdad80b4aa0 2023-11-08T08:21:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadf4e3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad80b5450 con 0x7fdad80c7eb0 2023-11-08T08:21:41.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadf4e3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad80c6e10 con 0x7fdac4003a00 2023-11-08T08:21:41.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 0x7fdad80b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 0x7fdad80b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47976/0 (socket says 172.21.15.116:47976) 2023-11-08T08:21:41.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 -- 172.21.15.116:0/898670833 learned_addr learned my addr 172.21.15.116:0/898670833 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadca7e700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdad80c7eb0 0x7fdad80c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadd27f700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4003a00 0x7fdad8142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 -- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4003a00 msgr2=0x7fdad8142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4003a00 0x7fdad8142900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 -- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdad80c7eb0 msgr2=0x7fdad80c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdad80c7eb0 0x7fdad80c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.467+0000 7fdadda80700 1 -- 172.21.15.116:0/898670833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad806f580 con 0x7fdad80b4aa0 2023-11-08T08:21:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.468+0000 7fdadda80700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 0x7fdad80b4f10 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdad806c4f0 tx=0x7fdad000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.468+0000 7fdacdffb700 1 -- 172.21.15.116:0/898670833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0003c10 con 0x7fdad80b4aa0 2023-11-08T08:21:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.468+0000 7fdacdffb700 1 -- 172.21.15.116:0/898670833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad0003d90 con 0x7fdad80b4aa0 2023-11-08T08:21:41.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.468+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad80cc130 con 0x7fdad80b4aa0 2023-11-08T08:21:41.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.468+0000 7fdacdffb700 1 -- 172.21.15.116:0/898670833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0005310 con 0x7fdad80b4aa0 2023-11-08T08:21:41.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.468+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad80cc410 con 0x7fdad80b4aa0 2023-11-08T08:21:41.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.469+0000 7fdacdffb700 1 -- 172.21.15.116:0/898670833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdad0005470 con 0x7fdad80b4aa0 2023-11-08T08:21:41.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.469+0000 7fdacdffb700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdabc0682f0 0x7fdabc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.470+0000 7fdacdffb700 1 -- 172.21.15.116:0/898670833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdad0085e40 con 0x7fdad80b4aa0 2023-11-08T08:21:41.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.470+0000 7fdadd27f700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdabc0682f0 0x7fdabc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.470+0000 7fdadd27f700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdabc0682f0 0x7fdabc06a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdad4001930 tx=0x7fdad400a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 Processor -- start 2023-11-08T08:21:41.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 -- start start 2023-11-08T08:21:41.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad78fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7d6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad790bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad70de90 con 0x55bbad7d5fc0 2023-11-08T08:21:41.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad67a1f0 con 0x55bbad7b86d0 2023-11-08T08:21:41.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bbad6e8e60 con 0x55bbad7ccb30 2023-11-08T08:21:41.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdadd27f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad78fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdadd27f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad78fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47982/0 (socket says 172.21.15.116:47982) 2023-11-08T08:21:41.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.472+0000 7fdadd27f700 1 -- 172.21.15.116:0/2014941830 learned_addr learned my addr 172.21.15.116:0/2014941830 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadca7e700 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7d6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadda80700 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad790bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadd27f700 1 -- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 msgr2=0x55bbad7d6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7d6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadd27f700 1 -- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 msgr2=0x55bbad790bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad790bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadd27f700 1 -- 172.21.15.116:0/2014941830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbad4d6840 con 0x55bbad7d5fc0 2023-11-08T08:21:41.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad78fa60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007ca0 tx=0x7fdad4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee37885fc6da1097 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdac2ffd700 1 -- 172.21.15.116:0/2014941830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4021070 con 0x55bbad7d5fc0 2023-11-08T08:21:41.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdac2ffd700 1 -- 172.21.15.116:0/2014941830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad402d070 con 0x55bbad7d5fc0 2023-11-08T08:21:41.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.473+0000 7fdac2ffd700 1 -- 172.21.15.116:0/2014941830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4005d70 con 0x55bbad7d5fc0 2023-11-08T08:21:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2014941830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 msgr2=0x55bbad78fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad78fa60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007ca0 tx=0x7fdad4013000 comp rx=0 tx=0).stop 2023-11-08T08:21:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2014941830 shutdown_connections 2023-11-08T08:21:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad790bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad78fa60 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2014941830 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7d6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2014941830 >> 172.21.15.116:0/2014941830 conn(0x55bbad785d90 msgr2=0x55bbad78ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2014941830 shutdown_connections 2023-11-08T08:21:41.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2014941830 wait complete. 2023-11-08T08:21:41.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 -- start start 2023-11-08T08:21:41.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.474+0000 7fdaeb682b80 1 Processor -- start 2023-11-08T08:21:41.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7e9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7e4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad7d3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad6e21d0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad4d7440 con 0x55bbad7b86d0 2023-11-08T08:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bbad67a3f0 con 0x55bbad7ccb30 2023-11-08T08:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdaeb682b80 10 client.0 _ll_register_callbacks cb 0x55bbad6804b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdadda80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad7d3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdadda80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad7d3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47986/0 (socket says 172.21.15.116:47986) 2023-11-08T08:21:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdadda80700 1 -- 172.21.15.116:0/2850398077 learned_addr learned my addr 172.21.15.116:0/2850398077 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.475+0000 7fdadca7e700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7e4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadda80700 1 -- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 msgr2=0x55bbad7e4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadda80700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7e4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadda80700 1 -- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 msgr2=0x55bbad7e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadda80700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadda80700 1 -- 172.21.15.116:0/2850398077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac40035b0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadda80700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad7d3ad0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x55bbad791fd0 tx=0x7fdad0055d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b4ff6b9b09423f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdab7fff700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdab7fff700 1 -- 172.21.15.116:0/2850398077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0055a70 con 0x55bbad7d5fc0 2023-11-08T08:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bbad6e60e0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdab7fff700 1 -- 172.21.15.116:0/2850398077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdad005e0b0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdab7fff700 1 -- 172.21.15.116:0/2850398077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdad00520a0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 10 client.6399 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:41.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55bbad5c3150 con 0x55bbad7d5fc0 2023-11-08T08:21:41.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdab7fff700 1 -- 172.21.15.116:0/2850398077 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bbad6e60e0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdab7fff700 1 -- 172.21.15.116:0/2850398077 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fdad0052200 con 0x55bbad7d5fc0 2023-11-08T08:21:41.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 10 client.6399 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:41.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 10 client.6399 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2023-11-08T08:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 20 client.6399 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7ccfa0 0x55bbad7cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 4 client.6399 mds_command: new command op to 6222 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbad531fa0 con 0x55bbad7ccfa0 2023-11-08T08:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.476+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7ccfa0 0x55bbad7cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdab7fff700 10 client.6399 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7ccfa0 0x55bbad7cd3a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=580f7880df243937 server_cookie=2c19c655eff28ac in_seq=0 out_seq=0 2023-11-08T08:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdab7fff700 1 -- 172.21.15.116:0/2850398077 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bbad531fa0 con 0x55bbad7ccfa0 2023-11-08T08:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdab7fff700 10 client.6399 handle_command_reply: tid=0 2023-11-08T08:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 client.6399 shutdown 2023-11-08T08:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 msgr2=0x55bbad7d3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad7d3ad0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x55bbad791fd0 tx=0x7fdad0055d10 comp rx=0 tx=0).stop 2023-11-08T08:21:41.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 shutdown_connections 2023-11-08T08:21:41.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7ccfa0 0x55bbad7cd3a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7d5fc0 0x55bbad7d3ad0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdab7fff700 0 client.6399 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2850398077 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7e4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.477+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 >> 172.21.15.116:0/2850398077 conn(0x55bbad785d90 msgr2=0x55bbad7b7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.478+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 shutdown_connections 2023-11-08T08:21:41.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.478+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2850398077 wait complete. 2023-11-08T08:21:41.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.478+0000 7fdaeb682b80 20 client.6399 trim_cache size 0 max 16384 2023-11-08T08:21:41.508 INFO:tasks.ceph.mds.d.smithi129.stderr:2023-11-08T08:21:41.501+0000 7f28d772a700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6366=0-4194304@1} caps={6366=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6366 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x556ba6acf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-08T08:21:41.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.494+0000 7fdaeb682b80 1 Processor -- start 2023-11-08T08:21:41.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.494+0000 7fdaeb682b80 1 -- start start 2023-11-08T08:21:41.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad785d90 0x55bbad7923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 0x55bbad7cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad7bbd10 con 0x55bbad7ccb30 2023-11-08T08:21:41.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad7bb830 con 0x55bbad7b86d0 2023-11-08T08:21:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bbad7b73a0 con 0x55bbad785d90 2023-11-08T08:21:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 0x55bbad7cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 0x55bbad7cf650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47990/0 (socket says 172.21.15.116:47990) 2023-11-08T08:21:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 -- 172.21.15.116:0/3749746340 learned_addr learned my addr 172.21.15.116:0/3749746340 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadd27f700 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad785d90 0x55bbad7923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadca7e700 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 -- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad785d90 msgr2=0x55bbad7923b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad785d90 0x55bbad7923b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 -- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 msgr2=0x55bbad7ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 -- 172.21.15.116:0/3749746340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbad6a9390 con 0x55bbad7ccb30 2023-11-08T08:21:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadca7e700 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdadda80700 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 0x55bbad7cf650 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fdad005de10 tx=0x7fdad006a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882358ccf0aae972 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdac37fe700 1 -- 172.21.15.116:0/3749746340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0055210 con 0x55bbad7ccb30 2023-11-08T08:21:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdac37fe700 1 -- 172.21.15.116:0/3749746340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad005e340 con 0x55bbad7ccb30 2023-11-08T08:21:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.495+0000 7fdac37fe700 1 -- 172.21.15.116:0/3749746340 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad005e0b0 con 0x55bbad7ccb30 2023-11-08T08:21:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 -- 172.21.15.116:0/3749746340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 msgr2=0x55bbad7cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 0x55bbad7cf650 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fdad005de10 tx=0x7fdad006a000 comp rx=0 tx=0).stop 2023-11-08T08:21:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 -- 172.21.15.116:0/3749746340 shutdown_connections 2023-11-08T08:21:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad7ccb30 0x55bbad7cf650 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad785d90 0x55bbad7923b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/3749746340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 -- 172.21.15.116:0/3749746340 >> 172.21.15.116:0/3749746340 conn(0x55bbad7b8100 msgr2=0x55bbad7e4930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 -- 172.21.15.116:0/3749746340 shutdown_connections 2023-11-08T08:21:41.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.496+0000 7fdaeb682b80 1 -- 172.21.15.116:0/3749746340 wait complete. 2023-11-08T08:21:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.497+0000 7fdaeb682b80 1 -- start start 2023-11-08T08:21:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.497+0000 7fdaeb682b80 1 Processor -- start 2023-11-08T08:21:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 0x55bbad832f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7acf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad835e80 con 0x55bbad785d90 2023-11-08T08:21:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbad7b5c80 con 0x55bbad7b86d0 2023-11-08T08:21:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bbad835c30 con 0x55bbad7ccb30 2023-11-08T08:21:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdaeb682b80 10 client.0 _ll_register_callbacks cb 0x55bbad6804b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 0x55bbad832f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 0x55bbad832f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48004/0 (socket says 172.21.15.116:48004) 2023-11-08T08:21:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 -- 172.21.15.116:0/2305727570 learned_addr learned my addr 172.21.15.116:0/2305727570 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadca7e700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadda80700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 -- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 msgr2=0x55bbad7acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7acf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 -- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 msgr2=0x55bbad7b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 -- 172.21.15.116:0/2305727570 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac40035b0 con 0x55bbad785d90 2023-11-08T08:21:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdadd27f700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 0x55bbad832f60 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c800 tx=0x7fdad4013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c4cd89be131e14 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.498+0000 7fdab7fff700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.499+0000 7fdab7fff700 1 -- 172.21.15.116:0/2305727570 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4038040 con 0x55bbad785d90 2023-11-08T08:21:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.499+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdad402d070 con 0x55bbad785d90 2023-11-08T08:21:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.499+0000 7fdab7fff700 1 -- 172.21.15.116:0/2305727570 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdad403f430 con 0x55bbad785d90 2023-11-08T08:21:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdab7fff700 1 -- 172.21.15.116:0/2305727570 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdad4004870 con 0x55bbad785d90 2023-11-08T08:21:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 10 client.6405 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55bbad537280 con 0x55bbad785d90 2023-11-08T08:21:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdab7fff700 1 -- 172.21.15.116:0/2305727570 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad402d070 con 0x55bbad785d90 2023-11-08T08:21:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdab7fff700 1 -- 172.21.15.116:0/2305727570 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fdad40049d0 con 0x55bbad785d90 2023-11-08T08:21:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 10 client.6405 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 10 client.6405 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2023-11-08T08:21:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 20 client.6405 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7d5fc0 0x55bbad7cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 4 client.6405 mds_command: new command op to 6222 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2023-11-08T08:21:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) v1 -- 0x55bbad5670f0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdadca7e700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7d5fc0 0x55bbad7cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdab7fff700 10 client.6405 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.500+0000 7fdadca7e700 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7d5fc0 0x55bbad7cbb50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34af180c53ec1274 server_cookie=9d56ddb8ee95fb0 in_seq=0 out_seq=0 2023-11-08T08:21:41.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdab7fff700 1 -- 172.21.15.116:0/2305727570 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bbad5670f0 con 0x55bbad7d5fc0 2023-11-08T08:21:41.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdab7fff700 10 client.6405 handle_command_reply: tid=0 2023-11-08T08:21:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 client.6405 shutdown 2023-11-08T08:21:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 msgr2=0x55bbad832f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 0x55bbad832f60 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c800 tx=0x7fdad4013070 comp rx=0 tx=0).stop 2023-11-08T08:21:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 shutdown_connections 2023-11-08T08:21:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55bbad7ccb30 0x55bbad7acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55bbad7d5fc0 0x55bbad7cbb50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbad785d90 0x55bbad832f60 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 --2- 172.21.15.116:0/2305727570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bbad7b86d0 0x55bbad7b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdab7fff700 0 client.6405 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 >> 172.21.15.116:0/2305727570 conn(0x55bbad7b8100 msgr2=0x55bbad7d1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 shutdown_connections 2023-11-08T08:21:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.501+0000 7fdaeb682b80 1 -- 172.21.15.116:0/2305727570 wait complete. 2023-11-08T08:21:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdaeb682b80 20 client.6405 trim_cache size 0 max 16384 2023-11-08T08:21:41.530 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:41.531 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2023-11-08T08:21:41.531 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "bf88b2fb-3e9c-4897-a8cb-d07a925c3840", 2023-11-08T08:21:41.531 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2023-11-08T08:21:41.531 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdabc0682f0 msgr2=0x7fdabc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdabc0682f0 0x7fdabc06a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdad4001930 tx=0x7fdad400a000 comp rx=0 tx=0).stop 2023-11-08T08:21:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 msgr2=0x7fdad80b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 0x7fdad80b4f10 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdad806c4f0 tx=0x7fdad000ee50 comp rx=0 tx=0).stop 2023-11-08T08:21:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 shutdown_connections 2023-11-08T08:21:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80b4aa0 0x7fdad80b4f10 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdac4003a00 0x7fdad8142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdabc0682f0 0x7fdabc06a7a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 --2- 172.21.15.116:0/898670833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdad80c7eb0 0x7fdad80c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 >> 172.21.15.116:0/898670833 conn(0x7fdad80cfc30 msgr2=0x7fdad815b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 shutdown_connections 2023-11-08T08:21:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.502+0000 7fdadf4e3700 1 -- 172.21.15.116:0/898670833 wait complete. 2023-11-08T08:21:41.535 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2023-11-08T08:21:41.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.656+0000 7fdd0fe8b700 1 Processor -- start 2023-11-08T08:21:41.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.656+0000 7fdd0fe8b700 1 -- start start 2023-11-08T08:21:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0fe8b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd0815c810 0x7fdd08160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0fe8b700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdd0806a270 0x7fdd0806a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0fe8b700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0806abb0 con 0x7fdd080718e0 2023-11-08T08:21:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0fe8b700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0815cbe0 con 0x7fdd0815c810 2023-11-08T08:21:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0fe8b700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd081607a0 con 0x7fdd0806a270 2023-11-08T08:21:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd0815c810 0x7fdd08160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0d426700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdd080718e0 0x7fdd0806c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55402/0 (socket says 172.21.15.116:55402) 2023-11-08T08:21:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd0815c810 0x7fdd08160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48016/0 (socket says 172.21.15.116:48016) 2023-11-08T08:21:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0d426700 1 -- 172.21.15.116:0/580416648 learned_addr learned my addr 172.21.15.116:0/580416648 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:41.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0e428700 1 --2- 172.21.15.116:0/580416648 >> v2:172.21.15.129:3300/0 conn(0x7fdd0806a270 0x7fdd0806a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 -- 172.21.15.116:0/580416648 >> v2:172.21.15.129:3300/0 conn(0x7fdd0806a270 msgr2=0x7fdd0806a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/580416648 >> v2:172.21.15.129:3300/0 conn(0x7fdd0806a270 0x7fdd0806a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 -- 172.21.15.116:0/580416648 >> v1:172.21.15.116:6789/0 conn(0x7fdd080718e0 legacy=0x7fdd0806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 -- 172.21.15.116:0/580416648 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08160a50 con 0x7fdd0815c810 2023-11-08T08:21:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0e428700 1 --2- 172.21.15.116:0/580416648 >> v2:172.21.15.129:3300/0 conn(0x7fdd0806a270 0x7fdd0806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0cc25700 1 -- 172.21.15.116:0/580416648 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246730059 0 0) 0x7fdd0806abb0 con 0x7fdd080718e0 2023-11-08T08:21:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.657+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/580416648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd0815c810 0x7fdd08160260 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fdd08071870 tx=0x7fdd04009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dc7ed67ea5c812 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0cc25700 1 -- 172.21.15.116:0/580416648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04011830 con 0x7fdd0815c810 2023-11-08T08:21:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0cc25700 1 -- 172.21.15.116:0/580416648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd04011990 con 0x7fdd0815c810 2023-11-08T08:21:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0cc25700 1 -- 172.21.15.116:0/580416648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04011d20 con 0x7fdd0815c810 2023-11-08T08:21:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/580416648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd0815c810 msgr2=0x7fdd08160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/580416648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd0815c810 0x7fdd08160260 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fdd08071870 tx=0x7fdd04009b20 comp rx=0 tx=0).stop 2023-11-08T08:21:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/580416648 shutdown_connections 2023-11-08T08:21:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/580416648 >> v2:172.21.15.129:3300/0 conn(0x7fdd0806a270 0x7fdd0806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/580416648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd0815c810 0x7fdd08160260 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/580416648 >> 172.21.15.116:0/580416648 conn(0x7fdd0806c010 msgr2=0x7fdd080cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/580416648 shutdown_connections 2023-11-08T08:21:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.658+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/580416648 wait complete. 2023-11-08T08:21:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.659+0000 7fdd0fe8b700 1 Processor -- start 2023-11-08T08:21:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.659+0000 7fdd0fe8b700 1 -- start start 2023-11-08T08:21:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0fe8b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd0806a270 0x7fdd080b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0fe8b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 0x7fdd080c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0fe8b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdd0815c810 0x7fdd080c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0fe8b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080afc60 con 0x7fdd080718e0 2023-11-08T08:21:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0fe8b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080b0630 con 0x7fdd0806a270 2023-11-08T08:21:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0fe8b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080af050 con 0x7fdd0815c810 2023-11-08T08:21:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 0x7fdd080c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 0x7fdd080c8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48030/0 (socket says 172.21.15.116:48030) 2023-11-08T08:21:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 -- 172.21.15.116:0/4093937123 learned_addr learned my addr 172.21.15.116:0/4093937123 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd0806a270 0x7fdd080b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0e428700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdd0815c810 0x7fdd080c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 -- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdd0815c810 msgr2=0x7fdd080c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdd0815c810 0x7fdd080c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 -- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd0806a270 msgr2=0x7fdd080b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd0806a270 0x7fdd080b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0d426700 1 -- 172.21.15.116:0/4093937123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd040094a0 con 0x7fdd080718e0 2023-11-08T08:21:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.660+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd0806a270 0x7fdd080b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.661+0000 7fdd0d426700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 0x7fdd080c8fd0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80011e0 tx=0x7fdcf800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.661+0000 7fdcfe7fc700 1 -- 172.21.15.116:0/4093937123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8004c80 con 0x7fdd080718e0 2023-11-08T08:21:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.661+0000 7fdcfe7fc700 1 -- 172.21.15.116:0/4093937123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8004e00 con 0x7fdd080718e0 2023-11-08T08:21:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.661+0000 7fdcfe7fc700 1 -- 172.21.15.116:0/4093937123 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8007310 con 0x7fdd080718e0 2023-11-08T08:21:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.661+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd08161010 con 0x7fdd080718e0 2023-11-08T08:21:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.661+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd08160d30 con 0x7fdd080718e0 2023-11-08T08:21:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.662+0000 7fdcfe7fc700 1 -- 172.21.15.116:0/4093937123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcf801a020 con 0x7fdd080718e0 2023-11-08T08:21:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.662+0000 7fdcfe7fc700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdcec068320 0x7fdcec06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.663+0000 7fdcfe7fc700 1 -- 172.21.15.116:0/4093937123 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdcf8085890 con 0x7fdd080718e0 2023-11-08T08:21:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.663+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdcec068320 0x7fdcec06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.663+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdcec068320 0x7fdcec06a7d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd040090b0 tx=0x7fdd0401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.664+0000 7fdd1c02ab80 1 Processor -- start 2023-11-08T08:21:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 -- start start 2023-11-08T08:21:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x557149194a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491d1ad0 0x5571491db330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x557149195b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557149112e30 con 0x5571491daf60 2023-11-08T08:21:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55714907f190 con 0x5571491d1ad0 2023-11-08T08:21:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5571490ede00 con 0x5571491bd670 2023-11-08T08:21:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd0dc27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x557149194a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd0dc27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x557149194a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48032/0 (socket says 172.21.15.116:48032) 2023-11-08T08:21:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd0dc27700 1 -- 172.21.15.116:0/31550292 learned_addr learned my addr 172.21.15.116:0/31550292 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd0e428700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x557149195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.665+0000 7fdd0d426700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491d1ad0 0x5571491db330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0dc27700 1 -- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 msgr2=0x557149195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x557149195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0dc27700 1 -- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491d1ad0 msgr2=0x5571491db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491d1ad0 0x5571491db330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0dc27700 1 -- 172.21.15.116:0/31550292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557148edb7e0 con 0x5571491daf60 2023-11-08T08:21:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0d426700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491d1ad0 0x5571491db330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0e428700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x557149195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x557149194a00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdd04011df0 tx=0x7fdd04010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b147141f9ddab2a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdceb7fe700 1 -- 172.21.15.116:0/31550292 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0402a070 con 0x5571491daf60 2023-11-08T08:21:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdceb7fe700 1 -- 172.21.15.116:0/31550292 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd040037a0 con 0x5571491daf60 2023-11-08T08:21:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.666+0000 7fdceb7fe700 1 -- 172.21.15.116:0/31550292 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04003950 con 0x5571491daf60 2023-11-08T08:21:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/31550292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 msgr2=0x557149194a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x557149194a00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdd04011df0 tx=0x7fdd04010000 comp rx=0 tx=0).stop 2023-11-08T08:21:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/31550292 shutdown_connections 2023-11-08T08:21:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x557149195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x557149194a00 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/31550292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491d1ad0 0x5571491db330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/31550292 >> 172.21.15.116:0/31550292 conn(0x55714918ad30 msgr2=0x557149193f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/31550292 shutdown_connections 2023-11-08T08:21:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/31550292 wait complete. 2023-11-08T08:21:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 -- start start 2023-11-08T08:21:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.667+0000 7fdd1c02ab80 1 Processor -- start 2023-11-08T08:21:41.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491d7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491ba890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491badd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571490e7170 con 0x5571491daf60 2023-11-08T08:21:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x557148edc3e0 con 0x5571491bd670 2023-11-08T08:21:41.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55714907f390 con 0x5571491d1ad0 2023-11-08T08:21:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd1c02ab80 10 client.0 _ll_register_callbacks cb 0x557149085450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd0e428700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491badd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd0e428700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491badd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48042/0 (socket says 172.21.15.116:48042) 2023-11-08T08:21:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd0e428700 1 -- 172.21.15.116:0/2235093974 learned_addr learned my addr 172.21.15.116:0/2235093974 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd0d426700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491ba890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.668+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491d7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd0e428700 1 -- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 msgr2=0x5571491ba890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd0e428700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491ba890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd0e428700 1 -- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 msgr2=0x5571491d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd0e428700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491d7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd0e428700 1 -- 172.21.15.116:0/2235093974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd08160a50 con 0x5571491daf60 2023-11-08T08:21:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd0e428700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491badd0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fdd000043a0 tx=0x7fdd000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06a759778f64d09 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdce8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdce8ff9700 1 -- 172.21.15.116:0/2235093974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00003ee0 con 0x5571491daf60 2023-11-08T08:21:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdce8ff9700 1 -- 172.21.15.116:0/2235093974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdd00007780 con 0x5571491daf60 2023-11-08T08:21:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571490eb080 con 0x5571491daf60 2023-11-08T08:21:41.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdce8ff9700 1 -- 172.21.15.116:0/2235093974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd00007940 con 0x5571491daf60 2023-11-08T08:21:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 10 client.6420 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x557148fc80f0 con 0x5571491daf60 2023-11-08T08:21:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdce8ff9700 1 -- 172.21.15.116:0/2235093974 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5571490eb080 con 0x5571491daf60 2023-11-08T08:21:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdce8ff9700 1 -- 172.21.15.116:0/2235093974 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fdd00014070 con 0x5571491daf60 2023-11-08T08:21:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 10 client.6420 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 10 client.6420 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2023-11-08T08:21:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 20 client.6420 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491b8cc0 0x5571491eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 4 client.6420 mds_command: new command op to 6222 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:21:41.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.669+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557148f36f40 con 0x5571491b8cc0 2023-11-08T08:21:41.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.670+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491b8cc0 0x5571491eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.670+0000 7fdce8ff9700 10 client.6420 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.670+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491b8cc0 0x5571491eb6b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55b3423580e668 server_cookie=cc4341db7627bdf6 in_seq=0 out_seq=0 2023-11-08T08:21:41.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.670+0000 7fdce8ff9700 1 -- 172.21.15.116:0/2235093974 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557148f36f40 con 0x5571491b8cc0 2023-11-08T08:21:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.670+0000 7fdce8ff9700 10 client.6420 handle_command_reply: tid=0 2023-11-08T08:21:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.670+0000 7fdd1c02ab80 1 client.6420 shutdown 2023-11-08T08:21:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 msgr2=0x5571491badd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491badd0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fdd000043a0 tx=0x7fdd000046b0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 shutdown_connections 2023-11-08T08:21:41.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491b8cc0 0x5571491eb6b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491badd0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdce8ff9700 0 client.6420 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491d7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/2235093974 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491ba890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 >> 172.21.15.116:0/2235093974 conn(0x55714918ad30 msgr2=0x55714918ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 shutdown_connections 2023-11-08T08:21:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/2235093974 wait complete. 2023-11-08T08:21:41.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.671+0000 7fdd1c02ab80 20 client.6420 trim_cache size 0 max 16384 2023-11-08T08:21:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.687+0000 7fdd1c02ab80 1 Processor -- start 2023-11-08T08:21:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.687+0000 7fdd1c02ab80 1 -- start start 2023-11-08T08:21:41.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x5571491bcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 0x5571491f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491daf60 0x5571491d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571491c3b30 con 0x5571491d1ad0 2023-11-08T08:21:41.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571491d77c0 con 0x5571491daf60 2023-11-08T08:21:41.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x557149196dc0 con 0x5571491bd670 2023-11-08T08:21:41.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 0x5571491f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 0x5571491f36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48052/0 (socket says 172.21.15.116:48052) 2023-11-08T08:21:41.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 -- 172.21.15.116:0/1775572319 learned_addr learned my addr 172.21.15.116:0/1775572319 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x5571491bcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0e428700 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491daf60 0x5571491d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 -- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 msgr2=0x5571491bcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x5571491bcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 -- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491daf60 msgr2=0x5571491d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491daf60 0x5571491d4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 -- 172.21.15.116:0/1775572319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571490ae330 con 0x5571491d1ad0 2023-11-08T08:21:41.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdd0d426700 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 0x5571491f36f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8052360 tx=0x7fdcf8055a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efd2c4262d62a58 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdcebfff700 1 -- 172.21.15.116:0/1775572319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8007470 con 0x5571491d1ad0 2023-11-08T08:21:41.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdcebfff700 1 -- 172.21.15.116:0/1775572319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8058040 con 0x5571491d1ad0 2023-11-08T08:21:41.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.688+0000 7fdcebfff700 1 -- 172.21.15.116:0/1775572319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8055350 con 0x5571491d1ad0 2023-11-08T08:21:41.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/1775572319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 msgr2=0x5571491f36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 0x5571491f36f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8052360 tx=0x7fdcf8055a70 comp rx=0 tx=0).stop 2023-11-08T08:21:41.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/1775572319 shutdown_connections 2023-11-08T08:21:41.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491daf60 0x5571491d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491bd670 0x5571491bcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/1775572319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491d1ad0 0x5571491f36f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/1775572319 >> 172.21.15.116:0/1775572319 conn(0x55714918ad30 msgr2=0x55714918ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/1775572319 shutdown_connections 2023-11-08T08:21:41.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/1775572319 wait complete. 2023-11-08T08:21:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 -- start start 2023-11-08T08:21:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.689+0000 7fdd1c02ab80 1 Processor -- start 2023-11-08T08:21:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491b9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491d0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571491e9f00 con 0x5571491daf60 2023-11-08T08:21:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571491b2740 con 0x5571491bd670 2023-11-08T08:21:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x5571491b2270 con 0x5571491d1ad0 2023-11-08T08:21:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd1c02ab80 10 client.0 _ll_register_callbacks cb 0x557149085450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0e428700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491d0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0dc27700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0d426700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0d426700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491b9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:36238/0 (socket says 172.21.15.116:36238) 2023-11-08T08:21:41.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0dc27700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491b1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:42974/0 (socket says 172.21.15.116:42974) 2023-11-08T08:21:41.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0e428700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491d0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48062/0 (socket says 172.21.15.116:48062) 2023-11-08T08:21:41.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.690+0000 7fdd0d426700 1 -- 172.21.15.116:0/3436194622 learned_addr learned my addr 172.21.15.116:0/3436194622 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0d426700 1 -- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 msgr2=0x5571491b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0d426700 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0d426700 1 -- 172.21.15.116:0/3436194622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 msgr2=0x5571491d0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0d426700 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491d0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0d426700 1 -- 172.21.15.116:0/3436194622 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd08160d30 con 0x5571491d1ad0 2023-11-08T08:21:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0e428700 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491d0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd0d426700 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491b9490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80541b0 tx=0x7fdcf8000f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e663870c86e42d92 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdce8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.129:3301/0 2023-11-08T08:21:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdce8ff9700 1 -- 172.21.15.116:0/3436194622 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8069080 con 0x5571491d1ad0 2023-11-08T08:21:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdce8ff9700 1 -- 172.21.15.116:0/3436194622 <== mon.2 v2:172.21.15.129:3301/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdcf80691e0 con 0x5571491d1ad0 2023-11-08T08:21:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557149197090 con 0x5571491d1ad0 2023-11-08T08:21:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdce8ff9700 1 -- 172.21.15.116:0/3436194622 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcf805d9b0 con 0x5571491d1ad0 2023-11-08T08:21:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 10 client.5363 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x557148f3c220 con 0x5571491d1ad0 2023-11-08T08:21:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdce8ff9700 1 -- 172.21.15.116:0/3436194622 <== mon.2 v2:172.21.15.129:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557149197090 con 0x5571491d1ad0 2023-11-08T08:21:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdce8ff9700 1 -- 172.21.15.116:0/3436194622 <== mon.2 v2:172.21.15.129:3301/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fdcf807f070 con 0x5571491d1ad0 2023-11-08T08:21:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 10 client.5363 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 10 client.5363 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2023-11-08T08:21:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 20 client.5363 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:41.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491d1450 0x5571492430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 4 client.5363 mds_command: new command op to 6222 tid=0[{"prefix": "scrub status"}] 2023-11-08T08:21:41.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.691+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fdd040037a0 con 0x5571491d1450 2023-11-08T08:21:41.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491d1450 0x5571492430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd0dc27700 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491d1450 0x5571492430a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a614b6349e26bf server_cookie=bd48a210e1f05b43 in_seq=0 out_seq=0 2023-11-08T08:21:41.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdce8ff9700 10 client.5363 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdce8ff9700 1 -- 172.21.15.116:0/3436194622 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fdd040037a0 con 0x5571491d1450 2023-11-08T08:21:41.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdce8ff9700 10 client.5363 handle_command_reply: tid=0 2023-11-08T08:21:41.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 client.5363 shutdown 2023-11-08T08:21:41.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 msgr2=0x5571491b9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491b9490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80541b0 tx=0x7fdcf8000f60 comp rx=0 tx=0).stop 2023-11-08T08:21:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 shutdown_connections 2023-11-08T08:21:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x5571491d1450 0x5571492430a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571491daf60 0x5571491d0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571491bd670 0x5571491b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 --2- 172.21.15.116:0/3436194622 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x5571491d1ad0 0x5571491b9490 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdce8ff9700 0 client.5363 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 >> 172.21.15.116:0/3436194622 conn(0x55714918ad30 msgr2=0x5571491d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 shutdown_connections 2023-11-08T08:21:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 1 -- 172.21.15.116:0/3436194622 wait complete. 2023-11-08T08:21:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.692+0000 7fdd1c02ab80 20 client.5363 trim_cache size 0 max 16384 2023-11-08T08:21:41.721 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:41.721 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2023-11-08T08:21:41.721 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2023-11-08T08:21:41.722 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdcec068320 msgr2=0x7fdcec06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdcec068320 0x7fdcec06a7d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd040090b0 tx=0x7fdd0401c040 comp rx=0 tx=0).stop 2023-11-08T08:21:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 msgr2=0x7fdd080c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 0x7fdd080c8fd0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80011e0 tx=0x7fdcf800cae0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 shutdown_connections 2023-11-08T08:21:41.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fdd0815c810 0x7fdd080c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fdcec068320 0x7fdcec06a7d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd0806a270 0x7fdd080b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 --2- 172.21.15.116:0/4093937123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd080718e0 0x7fdd080c8fd0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 >> 172.21.15.116:0/4093937123 conn(0x7fdd0806c010 msgr2=0x7fdd080b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 shutdown_connections 2023-11-08T08:21:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.693+0000 7fdd0fe8b700 1 -- 172.21.15.116:0/4093937123 wait complete. 2023-11-08T08:21:41.725 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-08T08:21:41.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.845+0000 7f9c9492e700 1 Processor -- start 2023-11-08T08:21:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.845+0000 7f9c9492e700 1 -- start start 2023-11-08T08:21:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c9492e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c90160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c9492e700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c900b1810 con 0x7f9c9015cc80 2023-11-08T08:21:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c9492e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c900b1af0 con 0x7f9c9006a270 2023-11-08T08:21:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c9492e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900b2490 con 0x7f9c9006e570 2023-11-08T08:21:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8dd9b700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9c9015cc80 0x7f9c9006de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55406/0 (socket says 172.21.15.116:55406) 2023-11-08T08:21:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/16098363 learned_addr learned my addr 172.21.15.116:0/16098363 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8ed9d700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c90160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827793039 0 0) 0x7f9c900b1810 con 0x7f9c9015cc80 2023-11-08T08:21:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c780035b0 con 0x7f9c9015cc80 2023-11-08T08:21:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503719879 0 0) 0x7f9c900b1af0 con 0x7f9c9006a270 2023-11-08T08:21:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c900b1810 con 0x7f9c9006a270 2023-11-08T08:21:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.846+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759196139 0 0) 0x7f9c780035b0 con 0x7f9c9015cc80 2023-11-08T08:21:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 msgr2=0x7f9c90160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c90160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 >> v1:172.21.15.129:6789/0 conn(0x7f9c9006a270 legacy=0x7f9c9006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900b1dd0 con 0x7f9c9015cc80 2023-11-08T08:21:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f9c80001670 con 0x7f9c9015cc80 2023-11-08T08:21:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 >> v1:172.21.15.116:6789/0 conn(0x7f9c9015cc80 legacy=0x7f9c9006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78003bc0 0x7f9c78003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c78004850 0x7f9c78004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900b2490 con 0x7f9c78004850 2023-11-08T08:21:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7800baf0 con 0x7f9c78003bc0 2023-11-08T08:21:41.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7800bcb0 con 0x7f9c9006e570 2023-11-08T08:21:41.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c78004850 0x7f9c78004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8e59c700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78003bc0 0x7f9c78003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8ed9d700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 msgr2=0x7f9c78005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c78005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78003bc0 msgr2=0x7f9c78003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:41.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78003bc0 0x7f9c78003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/16098363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780035b0 con 0x7f9c78004850 2023-11-08T08:21:41.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8ed9d700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c9006e570 0x7f9c78005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:41.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c78004850 0x7f9c78004cc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9c80007cb0 tx=0x7f9c78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab5da8587654134 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000cd50 con 0x7f9c78004850 2023-11-08T08:21:41.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c80007b10 con 0x7f9c78004850 2023-11-08T08:21:41.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.847+0000 7f9c8d59a700 1 -- 172.21.15.116:0/16098363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80005ba0 con 0x7f9c78004850 2023-11-08T08:21:41.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c9492e700 1 -- 172.21.15.116:0/16098363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c78004850 msgr2=0x7f9c78004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c9492e700 1 --2- 172.21.15.116:0/16098363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c78004850 0x7f9c78004cc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9c80007cb0 tx=0x7f9c78005880 comp rx=0 tx=0).stop 2023-11-08T08:21:41.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c8ed9d700 1 -- 172.21.15.116:0/16098363 reap_dead start 2023-11-08T08:21:41.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c9492e700 1 -- 172.21.15.116:0/16098363 shutdown_connections 2023-11-08T08:21:41.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c9492e700 1 -- 172.21.15.116:0/16098363 >> 172.21.15.116:0/16098363 conn(0x7f9c9006c010 msgr2=0x7f9c900cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:41.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c9492e700 1 -- 172.21.15.116:0/16098363 shutdown_connections 2023-11-08T08:21:41.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.848+0000 7f9c9492e700 1 -- 172.21.15.116:0/16098363 wait complete. 2023-11-08T08:21:41.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.849+0000 7f9c9492e700 1 Processor -- start 2023-11-08T08:21:41.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.849+0000 7f9c9492e700 1 -- start start 2023-11-08T08:21:41.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c9492e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78004850 0x7f9c900cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c9492e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 0x7f9c900c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c9492e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c901496e0 0x7f9c9015ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c9492e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900cc520 con 0x7f9c90149f10 2023-11-08T08:21:41.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c9492e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9014f230 con 0x7f9c78004850 2023-11-08T08:21:41.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c9492e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900ca450 con 0x7f9c901496e0 2023-11-08T08:21:41.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c8dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 0x7f9c900c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c8dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 0x7f9c900c6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48092/0 (socket says 172.21.15.116:48092) 2023-11-08T08:21:41.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/4259321197 learned_addr learned my addr 172.21.15.116:0/4259321197 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:41.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c8e59c700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78004850 0x7f9c900cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c8ed9d700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c901496e0 0x7f9c9015ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.850+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c901496e0 msgr2=0x7f9c9015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c901496e0 0x7f9c9015ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78004850 msgr2=0x7f9c900cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78004850 0x7f9c900cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c8dd9b700 1 -- 172.21.15.116:0/4259321197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9015daa0 con 0x7f9c90149f10 2023-11-08T08:21:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c8e59c700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78004850 0x7f9c900cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:41.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c8dd9b700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 0x7f9c900c6dc0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c800 tx=0x7f9c90153d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80025070 con 0x7f9c90149f10 2023-11-08T08:21:41.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8000cd60 con 0x7f9c90149f10 2023-11-08T08:21:41.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c900b20b0 con 0x7f9c90149f10 2023-11-08T08:21:41.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80001650 con 0x7f9c90149f10 2023-11-08T08:21:41.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.851+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9015dc30 con 0x7f9c90149f10 2023-11-08T08:21:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.852+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9014f560 con 0x7f9c90149f10 2023-11-08T08:21:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.852+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c80004870 con 0x7f9c90149f10 2023-11-08T08:21:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.852+0000 7f9c7effd700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9c700682f0 0x7f9c7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.853+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9c8008e3d0 con 0x7f9c90149f10 2023-11-08T08:21:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.853+0000 7f9c8e59c700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9c700682f0 0x7f9c7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.853+0000 7f9c8e59c700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9c700682f0 0x7f9c7006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9c84007fd0 tx=0x7f9c84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:41.857+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c8005ac30 con 0x7f9c90149f10 2023-11-08T08:21:42.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.077+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c9014f3a0 con 0x7f9c90149f10 2023-11-08T08:21:42.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.078+0000 7f9c7effd700 1 -- 172.21.15.116:0/4259321197 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4661 (secure 0 0 0) 0x7f9c9014f3a0 con 0x7f9c90149f10 2023-11-08T08:21:42.080 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:42.080 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:35.475147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:42.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9c700682f0 msgr2=0x7f9c7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9c700682f0 0x7f9c7006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9c84007fd0 tx=0x7f9c84009040 comp rx=0 tx=0).stop 2023-11-08T08:21:42.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 msgr2=0x7f9c900c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 0x7f9c900c6dc0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c800 tx=0x7f9c90153d30 comp rx=0 tx=0).stop 2023-11-08T08:21:42.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 shutdown_connections 2023-11-08T08:21:42.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9c901496e0 0x7f9c9015ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9c700682f0 0x7f9c7006a7a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c78004850 0x7f9c900cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 --2- 172.21.15.116:0/4259321197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c90149f10 0x7f9c900c6dc0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 >> 172.21.15.116:0/4259321197 conn(0x7f9c9006c010 msgr2=0x7f9c900c65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 shutdown_connections 2023-11-08T08:21:42.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.080+0000 7f9c9492e700 1 -- 172.21.15.116:0/4259321197 wait complete. 2023-11-08T08:21:42.084 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2023-11-08T08:21:42.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d damage ls --format=json-pretty 2023-11-08T08:21:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.214+0000 7f6604ce6700 1 Processor -- start 2023-11-08T08:21:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.214+0000 7f6604ce6700 1 -- start start 2023-11-08T08:21:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f6604ce6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f6600160260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f6604ce6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f660006a270 0x7f660006a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f6604ce6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f660006ac10 con 0x7f66000718e0 2023-11-08T08:21:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f6604ce6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f660015cbe0 con 0x7f660006a270 2023-11-08T08:21:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f6604ce6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66001607a0 con 0x7f660015c810 2023-11-08T08:21:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f660006a270 0x7f660006a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f660006a270 0x7f660006a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48102/0 (socket says 172.21.15.116:48102) 2023-11-08T08:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 -- 172.21.15.116:0/2048297178 learned_addr learned my addr 172.21.15.116:0/2048297178 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fe59c700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f6600160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 -- 172.21.15.116:0/2048297178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 msgr2=0x7f6600160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f6600160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 -- 172.21.15.116:0/2048297178 >> v1:172.21.15.129:6789/0 conn(0x7f66000718e0 legacy=0x7f660006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 -- 172.21.15.116:0/2048297178 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600160a50 con 0x7f660006a270 2023-11-08T08:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fe59c700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f6600160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fed9d700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f660006a270 0x7f660006a6d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009e50 tx=0x7f65f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cee6eb6dfd10d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fd59a700 1 -- 172.21.15.116:0/2048297178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f800e070 con 0x7f660006a270 2023-11-08T08:21:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fd59a700 1 -- 172.21.15.116:0/2048297178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f800b9c0 con 0x7f660006a270 2023-11-08T08:21:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.215+0000 7f65fd59a700 1 -- 172.21.15.116:0/2048297178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f800bd70 con 0x7f660006a270 2023-11-08T08:21:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 -- 172.21.15.116:0/2048297178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f660006a270 msgr2=0x7f660006a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f660006a270 0x7f660006a6d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009e50 tx=0x7f65f800b330 comp rx=0 tx=0).stop 2023-11-08T08:21:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 -- 172.21.15.116:0/2048297178 shutdown_connections 2023-11-08T08:21:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f660006a270 0x7f660006a6d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 --2- 172.21.15.116:0/2048297178 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f6600160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 -- 172.21.15.116:0/2048297178 >> 172.21.15.116:0/2048297178 conn(0x7f660006c010 msgr2=0x7f66000cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 -- 172.21.15.116:0/2048297178 shutdown_connections 2023-11-08T08:21:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.216+0000 7f6604ce6700 1 -- 172.21.15.116:0/2048297178 wait complete. 2023-11-08T08:21:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.217+0000 7f6604ce6700 1 Processor -- start 2023-11-08T08:21:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.217+0000 7f6604ce6700 1 -- start start 2023-11-08T08:21:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f6604ce6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660006a270 0x7f660014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f6604ce6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 0x7f6600148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f6604ce6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f660015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f6604ce6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66000c53d0 con 0x7f66000718e0 2023-11-08T08:21:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f6604ce6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001502f0 con 0x7f660006a270 2023-11-08T08:21:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f6604ce6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66001486c0 con 0x7f660015c810 2023-11-08T08:21:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 0x7f6600148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 0x7f6600148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48110/0 (socket says 172.21.15.116:48110) 2023-11-08T08:21:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1240846338 learned_addr learned my addr 172.21.15.116:0/1240846338 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fe59c700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660006a270 0x7f660014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fed9d700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f660015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 msgr2=0x7f660015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f660015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660006a270 msgr2=0x7f660014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660006a270 0x7f660014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.218+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1240846338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f8009b30 con 0x7f66000718e0 2023-11-08T08:21:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.219+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 0x7f6600148180 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f65f000c4e0 tx=0x7f65f000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.219+0000 7f65eeffd700 1 -- 172.21.15.116:0/1240846338 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f00052d0 con 0x7f66000718e0 2023-11-08T08:21:42.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.219+0000 7f65eeffd700 1 -- 172.21.15.116:0/1240846338 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f0005430 con 0x7f66000718e0 2023-11-08T08:21:42.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.219+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6600161010 con 0x7f66000718e0 2023-11-08T08:21:42.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.219+0000 7f65eeffd700 1 -- 172.21.15.116:0/1240846338 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f00038c0 con 0x7f66000718e0 2023-11-08T08:21:42.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.219+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6600160d30 con 0x7f66000718e0 2023-11-08T08:21:42.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.220+0000 7f65eeffd700 1 -- 172.21.15.116:0/1240846338 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65f0019030 con 0x7f66000718e0 2023-11-08T08:21:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.220+0000 7f65eeffd700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65e0068320 0x7f65e006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.221+0000 7f65eeffd700 1 -- 172.21.15.116:0/1240846338 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f65f0084fc0 con 0x7f66000718e0 2023-11-08T08:21:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.221+0000 7f65fe59c700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65e0068320 0x7f65e006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.221+0000 7f65fe59c700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65e0068320 0x7f65e006a7d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f660006abe0 tx=0x7f65f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.222+0000 7f6610e85b80 1 Processor -- start 2023-11-08T08:21:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 -- start start 2023-11-08T08:21:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f113572a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f113573bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1134f0e80 con 0x55f1135afb20 2023-11-08T08:21:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11345d1e0 con 0x55f1135b8fb0 2023-11-08T08:21:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.223+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1134cbe50 con 0x55f11359b6c0 2023-11-08T08:21:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48114/0 (socket says 172.21.15.116:48114) 2023-11-08T08:21:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3282303380 learned_addr learned my addr 172.21.15.116:0/3282303380 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fe59c700 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f113572a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fed9d700 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f113573bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 msgr2=0x55f113573bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f113573bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 msgr2=0x55f113572a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f113572a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3282303380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1132b9830 con 0x55f1135afb20 2023-11-08T08:21:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b9380 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f65f00550f0 tx=0x7f65f0055120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cb133e1f8f5756 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65dffff700 1 -- 172.21.15.116:0/3282303380 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f005b0d0 con 0x55f1135afb20 2023-11-08T08:21:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65dffff700 1 -- 172.21.15.116:0/3282303380 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f0053880 con 0x55f1135afb20 2023-11-08T08:21:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.224+0000 7f65dffff700 1 -- 172.21.15.116:0/3282303380 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f005d2e0 con 0x55f1135afb20 2023-11-08T08:21:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 -- 172.21.15.116:0/3282303380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 msgr2=0x55f1135b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b9380 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f65f00550f0 tx=0x7f65f0055120 comp rx=0 tx=0).stop 2023-11-08T08:21:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 -- 172.21.15.116:0/3282303380 shutdown_connections 2023-11-08T08:21:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f113573bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f113572a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3282303380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b9380 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 -- 172.21.15.116:0/3282303380 >> 172.21.15.116:0/3282303380 conn(0x55f113568d80 msgr2=0x55f113571f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 -- 172.21.15.116:0/3282303380 shutdown_connections 2023-11-08T08:21:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 -- 172.21.15.116:0/3282303380 wait complete. 2023-11-08T08:21:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.225+0000 7f6610e85b80 1 -- start start 2023-11-08T08:21:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 Processor -- start 2023-11-08T08:21:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f1135b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f1135b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1134c51c0 con 0x55f1135afb20 2023-11-08T08:21:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1132ba430 con 0x55f1135b8fb0 2023-11-08T08:21:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.226+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f11345d3e0 con 0x55f11359b6c0 2023-11-08T08:21:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f6610e85b80 10 client.0 _ll_register_callbacks cb 0x55f1134634a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48120/0 (socket says 172.21.15.116:48120) 2023-11-08T08:21:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1546039228 learned_addr learned my addr 172.21.15.116:0/1546039228 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fe59c700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f1135b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fed9d700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f1135b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 msgr2=0x55f1135b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f1135b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 msgr2=0x55f1135b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f1135b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 -- 172.21.15.116:0/1546039228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6600160d30 con 0x55f1135afb20 2023-11-08T08:21:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4b40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f65f0007be0 tx=0x7f65f0007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374b0c3e62f64195 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65dd7fa700 1 -- 172.21.15.116:0/1546039228 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f005b0d0 con 0x55f1135afb20 2023-11-08T08:21:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65dd7fa700 1 -- 172.21.15.116:0/1546039228 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f65f005d2e0 con 0x55f1135afb20 2023-11-08T08:21:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1134c90d0 con 0x55f1135afb20 2023-11-08T08:21:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65dd7fa700 1 -- 172.21.15.116:0/1546039228 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65f0083380 con 0x55f1135afb20 2023-11-08T08:21:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f6610e85b80 10 client.6444 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55f1133a6140 con 0x55f1135afb20 2023-11-08T08:21:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.227+0000 7f65dd7fa700 1 -- 172.21.15.116:0/1546039228 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f1134c90d0 con 0x55f1135afb20 2023-11-08T08:21:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f65dd7fa700 1 -- 172.21.15.116:0/1546039228 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f65f00512d0 con 0x55f1135afb20 2023-11-08T08:21:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f6610e85b80 10 client.6444 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f6610e85b80 10 client.6444 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-08T08:21:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f6610e85b80 20 client.6444 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f6610e85b80 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135b22d0 0x55f1135b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f6610e85b80 4 client.6444 mds_command: new command op to 6222 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-08T08:21:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f113314f90 con 0x55f1135b22d0 2023-11-08T08:21:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f65fe59c700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135b22d0 0x55f1135b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f65dd7fa700 10 client.6444 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.228+0000 7f65fe59c700 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135b22d0 0x55f1135b26a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=487da87e418c7f60 server_cookie=b8096cfa05813ee in_seq=0 out_seq=0 2023-11-08T08:21:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f65dd7fa700 1 -- 172.21.15.116:0/1546039228 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f113314f90 con 0x55f1135b22d0 2023-11-08T08:21:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f65dd7fa700 10 client.6444 handle_command_reply: tid=0 2023-11-08T08:21:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 client.6444 shutdown 2023-11-08T08:21:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 msgr2=0x55f1135b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4b40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f65f0007be0 tx=0x7f65f0007b70 comp rx=0 tx=0).stop 2023-11-08T08:21:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 shutdown_connections 2023-11-08T08:21:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1135b8fb0 0x55f1135b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f11359b6c0 0x55f1135b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135b22d0 0x55f1135b26a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 --2- 172.21.15.116:0/1546039228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4b40 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f65dd7fa700 0 client.6444 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 >> 172.21.15.116:0/1546039228 conn(0x55f113568d80 msgr2=0x55f11359ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 shutdown_connections 2023-11-08T08:21:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 1 -- 172.21.15.116:0/1546039228 wait complete. 2023-11-08T08:21:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.229+0000 7f6610e85b80 20 client.6444 trim_cache size 0 max 16384 2023-11-08T08:21:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 Processor -- start 2023-11-08T08:21:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 -- start start 2023-11-08T08:21:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f1135b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135b8fb0 0x55f1135b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1135a2140 con 0x55f1135afb20 2023-11-08T08:21:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11356e420 con 0x55f11359b6c0 2023-11-08T08:21:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f11359c950 con 0x55f1135b8fb0 2023-11-08T08:21:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f65fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48132/0 (socket says 172.21.15.116:48132) 2023-11-08T08:21:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.242+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3671903227 learned_addr learned my addr 172.21.15.116:0/3671903227 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3671903227 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135b8fb0 msgr2=0x55f1135b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135b8fb0 0x55f1135b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3671903227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 msgr2=0x55f1135b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f1135b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65fdd9b700 1 -- 172.21.15.116:0/3671903227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f11348c380 con 0x55f1135afb20 2023-11-08T08:21:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4a80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f65f000cca0 tx=0x7f65f006a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a8054d7f49de58 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65ecff9700 1 -- 172.21.15.116:0/3671903227 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f005b0d0 con 0x55f1135afb20 2023-11-08T08:21:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65ecff9700 1 -- 172.21.15.116:0/3671903227 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f000e040 con 0x55f1135afb20 2023-11-08T08:21:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f65ecff9700 1 -- 172.21.15.116:0/3671903227 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f005d300 con 0x55f1135afb20 2023-11-08T08:21:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 -- 172.21.15.116:0/3671903227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 msgr2=0x55f1135b4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4a80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f65f000cca0 tx=0x7f65f006a000 comp rx=0 tx=0).stop 2023-11-08T08:21:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 -- 172.21.15.116:0/3671903227 shutdown_connections 2023-11-08T08:21:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135b8fb0 0x55f1135b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f1135b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 --2- 172.21.15.116:0/3671903227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135afb20 0x55f1135b4a80 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.243+0000 7f6610e85b80 1 -- 172.21.15.116:0/3671903227 >> 172.21.15.116:0/3671903227 conn(0x55f113568d80 msgr2=0x55f11359ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.244+0000 7f6610e85b80 1 -- 172.21.15.116:0/3671903227 shutdown_connections 2023-11-08T08:21:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.244+0000 7f6610e85b80 1 -- 172.21.15.116:0/3671903227 wait complete. 2023-11-08T08:21:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.244+0000 7f6610e85b80 1 -- start start 2023-11-08T08:21:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.244+0000 7f6610e85b80 1 Processor -- start 2023-11-08T08:21:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f11361d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135afb20 0x55f1135a1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 0x55f1135a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11361db10 con 0x55f1135b8fb0 2023-11-08T08:21:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1135b5660 con 0x55f11359b6c0 2023-11-08T08:21:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1135d75d0 con 0x55f1135afb20 2023-11-08T08:21:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f6610e85b80 10 client.0 _ll_register_callbacks cb 0x55f1134634a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-08T08:21:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 0x55f1135a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 0x55f1135a1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45864/0 (socket says 172.21.15.116:45864) 2023-11-08T08:21:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fe59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f11361d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 -- 172.21.15.116:0/2215089421 learned_addr learned my addr 172.21.15.116:0/2215089421 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fdd9b700 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135afb20 0x55f1135a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 -- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135afb20 msgr2=0x55f1135a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135afb20 0x55f1135a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 -- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 msgr2=0x55f11361d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f11361d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 -- 172.21.15.116:0/2215089421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1133a6140 con 0x55f1135b8fb0 2023-11-08T08:21:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65fed9d700 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 0x55f1135a1a90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f65f8003c50 tx=0x7f65f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f2d01924ad9a66 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.245+0000 7f65dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-08T08:21:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f65dd7fa700 1 -- 172.21.15.116:0/2215089421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8004240 con 0x55f1135b8fb0 2023-11-08T08:21:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f65dd7fa700 1 -- 172.21.15.116:0/2215089421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f65f80043a0 con 0x55f1135b8fb0 2023-11-08T08:21:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f11334a0e0 con 0x55f1135b8fb0 2023-11-08T08:21:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f65dd7fa700 1 -- 172.21.15.116:0/2215089421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65f8004560 con 0x55f1135b8fb0 2023-11-08T08:21:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 10 client.6450 fetch_fsmap learned FSMap version 32 2023-11-08T08:21:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55f11331a270 con 0x55f1135b8fb0 2023-11-08T08:21:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f65dd7fa700 1 -- 172.21.15.116:0/2215089421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f11334a0e0 con 0x55f1135b8fb0 2023-11-08T08:21:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f65dd7fa700 1 -- 172.21.15.116:0/2215089421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f65f8019070 con 0x55f1135b8fb0 2023-11-08T08:21:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 10 client.6450 fetch_fsmap finished waiting for FSMap version 32 2023-11-08T08:21:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 10 client.6450 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-08T08:21:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 20 client.6450 populate_metadata read hostname 'smithi116' 2023-11-08T08:21:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135bc5c0 0x55f1135bc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 4 client.6450 mds_command: new command op to 6222 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2023-11-08T08:21:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 --> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55f113407460 con 0x55f1135bc5c0 2023-11-08T08:21:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.246+0000 7f65fe59c700 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135bc5c0 0x55f1135bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f65dd7fa700 10 client.6450 ms_handle_connect on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f65fe59c700 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135bc5c0 0x55f1135bc990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=268802a63cac60ce server_cookie=3cb488ca6e2feb9a in_seq=0 out_seq=0 2023-11-08T08:21:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f65dd7fa700 1 -- 172.21.15.116:0/2215089421 <== mds.0 v2:172.21.15.129:6833/4191005747 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+141 (crc 0 0 0) 0x55f113407460 con 0x55f1135bc5c0 2023-11-08T08:21:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f65dd7fa700 10 client.6450 handle_command_reply: tid=0 2023-11-08T08:21:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 client.6450 shutdown 2023-11-08T08:21:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 msgr2=0x55f1135a1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 0x55f1135a1a90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f65f8003c50 tx=0x7f65f8004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 shutdown_connections 2023-11-08T08:21:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:6833/4191005747,v1:172.21.15.129:6835/4191005747] conn(0x55f1135bc5c0 0x55f1135bc990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1135b8fb0 0x55f1135a1a90 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f65dd7fa700 0 client.6450 ms_handle_reset on v2:172.21.15.129:6833/4191005747 2023-11-08T08:21:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f11359b6c0 0x55f11361d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 --2- 172.21.15.116:0/2215089421 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x55f1135afb20 0x55f1135a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 >> 172.21.15.116:0/2215089421 conn(0x55f113568d80 msgr2=0x55f113599e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 shutdown_connections 2023-11-08T08:21:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 1 -- 172.21.15.116:0/2215089421 wait complete. 2023-11-08T08:21:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6610e85b80 20 client.6450 trim_cache size 0 max 16384 2023-11-08T08:21:42.275 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:42.276 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-08T08:21:42.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-08T08:21:42.276 INFO:teuthology.orchestra.run.smithi116.stdout: "damage_type": "backtrace", 2023-11-08T08:21:42.276 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 1139817786, 2023-11-08T08:21:42.277 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1099511627777, 2023-11-08T08:21:42.277 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/dir_test/file" 2023-11-08T08:21:42.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-08T08:21:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-08T08:21:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65e0068320 msgr2=0x7f65e006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6604ce6700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65e0068320 0x7f65e006a7d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f660006abe0 tx=0x7f65f4008040 comp rx=0 tx=0).stop 2023-11-08T08:21:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 msgr2=0x7f6600148180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.247+0000 7f6604ce6700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 0x7f6600148180 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f65f000c4e0 tx=0x7f65f000c8a0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 shutdown_connections 2023-11-08T08:21:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f660015c810 0x7f660015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f65e0068320 0x7f65e006a7d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660006a270 0x7f660014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 --2- 172.21.15.116:0/1240846338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66000718e0 0x7f6600148180 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 >> 172.21.15.116:0/1240846338 conn(0x7f660006c010 msgr2=0x7f6600155bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 shutdown_connections 2023-11-08T08:21:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.248+0000 7f6604ce6700 1 -- 172.21.15.116:0/1240846338 wait complete. 2023-11-08T08:21:42.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-08T08:21:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.401+0000 7f505bf7c700 1 Processor -- start 2023-11-08T08:21:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.401+0000 7f505bf7c700 1 -- start start 2023-11-08T08:21:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.402+0000 7f505bf7c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 0x7f505415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.402+0000 7f505bf7c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505413fc90 con 0x7f5054142db0 2023-11-08T08:21:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.402+0000 7f505bf7c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505413f480 con 0x7f50540b1770 2023-11-08T08:21:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.402+0000 7f505bf7c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505413f6d0 con 0x7f50540ce040 2023-11-08T08:21:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.402+0000 7f5059517700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5054142db0 0x7f505413ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39418/0 (socket says 172.21.15.116:39418) 2023-11-08T08:21:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.402+0000 7f5059517700 1 -- 172.21.15.116:0/3249508448 learned_addr learned my addr 172.21.15.116:0/3249508448 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225098040 0 0) 0x7f505413fc90 con 0x7f5054142db0 2023-11-08T08:21:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50400035b0 con 0x7f5054142db0 2023-11-08T08:21:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5059d18700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 0x7f505415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568073122 0 0) 0x7f50400035b0 con 0x7f5054142db0 2023-11-08T08:21:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 msgr2=0x7f505415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 0x7f505415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 >> v1:172.21.15.129:6789/0 conn(0x7f50540b1770 legacy=0x7f505413cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505413f9b0 con 0x7f5054142db0 2023-11-08T08:21:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f5050001670 con 0x7f5054142db0 2023-11-08T08:21:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 >> v1:172.21.15.116:6789/0 conn(0x7f5054142db0 legacy=0x7f505413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 0x7f5040003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5040004020 0x7f5040004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5040004d90 0x7f5040005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505413f6d0 con 0x7f5040004d90 2023-11-08T08:21:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504000bb20 con 0x7f5040004020 2023-11-08T08:21:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504000bce0 con 0x7f50540ce040 2023-11-08T08:21:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f505a519700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5040004d90 0x7f5040005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5059d18700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 0x7f5040003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f5059517700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5040004020 0x7f5040004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f505a519700 1 -- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 msgr2=0x7f5040003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f505a519700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540ce040 0x7f5040003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f505a519700 1 -- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5040004020 msgr2=0x7f5040004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f505a519700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5040004020 0x7f5040004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.403+0000 7f505a519700 1 -- 172.21.15.116:0/3249508448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50400035b0 con 0x7f5040004d90 2023-11-08T08:21:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505a519700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5040004d90 0x7f5040005200 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f504c00c290 tx=0x7f5040005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb818a06fd174f24 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c001e70 con 0x7f5040004d90 2023-11-08T08:21:42.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504c00ce50 con 0x7f5040004d90 2023-11-08T08:21:42.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f5058d16700 1 -- 172.21.15.116:0/3249508448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c012660 con 0x7f5040004d90 2023-11-08T08:21:42.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505bf7c700 1 -- 172.21.15.116:0/3249508448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5040004d90 msgr2=0x7f5040005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505bf7c700 1 --2- 172.21.15.116:0/3249508448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5040004d90 0x7f5040005200 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f504c00c290 tx=0x7f5040005880 comp rx=0 tx=0).stop 2023-11-08T08:21:42.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505a519700 1 -- 172.21.15.116:0/3249508448 reap_dead start 2023-11-08T08:21:42.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505bf7c700 1 -- 172.21.15.116:0/3249508448 shutdown_connections 2023-11-08T08:21:42.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505bf7c700 1 -- 172.21.15.116:0/3249508448 >> 172.21.15.116:0/3249508448 conn(0x7f505406c010 msgr2=0x7f50540cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505bf7c700 1 -- 172.21.15.116:0/3249508448 shutdown_connections 2023-11-08T08:21:42.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.404+0000 7f505bf7c700 1 -- 172.21.15.116:0/3249508448 wait complete. 2023-11-08T08:21:42.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.405+0000 7f505bf7c700 1 Processor -- start 2023-11-08T08:21:42.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.405+0000 7f505bf7c700 1 -- start start 2023-11-08T08:21:42.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505bf7c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 0x7f5054159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505bf7c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50541f1f50 0x7f50540cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505bf7c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540cb710 0x7f50540c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505bf7c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50540c37b0 con 0x7f50541f1b80 2023-11-08T08:21:42.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505bf7c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50540cbb90 con 0x7f50541f1f50 2023-11-08T08:21:42.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505bf7c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50540b2cc0 con 0x7f50540cb710 2023-11-08T08:21:42.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f5059d18700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 0x7f5054159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f5059d18700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 0x7f5054159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45880/0 (socket says 172.21.15.116:45880) 2023-11-08T08:21:42.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f5059d18700 1 -- 172.21.15.116:0/2824610436 learned_addr learned my addr 172.21.15.116:0/2824610436 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f5059517700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50541f1f50 0x7f50540cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.406+0000 7f505a519700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540cb710 0x7f50540c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f5059d18700 1 -- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540cb710 msgr2=0x7f50540c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f5059d18700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540cb710 0x7f50540c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f5059d18700 1 -- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50541f1f50 msgr2=0x7f50540cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f5059d18700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50541f1f50 0x7f50540cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f5059d18700 1 -- 172.21.15.116:0/2824610436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505415d3e0 con 0x7f50541f1b80 2023-11-08T08:21:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f5059d18700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 0x7f5054159c90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f50480116c0 tx=0x7f5048011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504800c070 con 0x7f50541f1b80 2023-11-08T08:21:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048015870 con 0x7f50541f1b80 2023-11-08T08:21:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505415d100 con 0x7f50541f1b80 2023-11-08T08:21:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048015c20 con 0x7f50541f1b80 2023-11-08T08:21:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.407+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50540ce410 con 0x7f50541f1b80 2023-11-08T08:21:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.408+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50540b3bb0 con 0x7f50541f1b80 2023-11-08T08:21:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.408+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5048025070 con 0x7f50541f1b80 2023-11-08T08:21:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.408+0000 7f50467fc700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50380682f0 0x7f503806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.409+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5048085f90 con 0x7f50541f1b80 2023-11-08T08:21:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.409+0000 7f5059517700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50380682f0 0x7f503806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.409+0000 7f5059517700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50380682f0 0x7f503806a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5050001050 tx=0x7f5050009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.413+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50480527f0 con 0x7f50541f1b80 2023-11-08T08:21:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.636+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f50540743f0 con 0x7f50541f1b80 2023-11-08T08:21:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.637+0000 7f50467fc700 1 -- 172.21.15.116:0/2824610436 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5048023040 con 0x7f50541f1b80 2023-11-08T08:21:42.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:42.639 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-08T08:21:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50380682f0 msgr2=0x7f503806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50380682f0 0x7f503806a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5050001050 tx=0x7f5050009040 comp rx=0 tx=0).stop 2023-11-08T08:21:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 msgr2=0x7f5054159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 0x7f5054159c90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f50480116c0 tx=0x7f5048011a80 comp rx=0 tx=0).stop 2023-11-08T08:21:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 shutdown_connections 2023-11-08T08:21:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f50540cb710 0x7f50540c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50541f1b80 0x7f5054159c90 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f50380682f0 0x7f503806a7a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 --2- 172.21.15.116:0/2824610436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50541f1f50 0x7f50540cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 >> 172.21.15.116:0/2824610436 conn(0x7f505406c010 msgr2=0x7f5054157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 shutdown_connections 2023-11-08T08:21:42.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.639+0000 7f505bf7c700 1 -- 172.21.15.116:0/2824610436 wait complete. 2023-11-08T08:21:42.652 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-08T08:21:42.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:21:42.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-08T08:21:42.738 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-08T08:21:42.739 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:42.739 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:42.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:42.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.917+0000 7fc01ecc9700 1 Processor -- start 2023-11-08T08:21:42.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.917+0000 7fc01ecc9700 1 -- start start 2023-11-08T08:21:42.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ecc9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ecc9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc01815e0b0 0x7fc01815e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ecc9700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc018151a50 con 0x7fc018070d90 2023-11-08T08:21:42.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ecc9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01815b440 con 0x7fc01815e0b0 2023-11-08T08:21:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ecc9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01806ab80 con 0x7fc01806a270 2023-11-08T08:21:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01d266700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc01815e0b0 0x7fc01815e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc017fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc018070d90 0x7fc0180711e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39426/0 (socket says 172.21.15.116:39426) 2023-11-08T08:21:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc017fff700 1 -- 172.21.15.116:0/1569378590 learned_addr learned my addr 172.21.15.116:0/1569378590 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ca65700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01d266700 1 -- 172.21.15.116:0/1569378590 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 msgr2=0x7fc01806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:42.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01d266700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01806a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01d266700 1 -- 172.21.15.116:0/1569378590 >> v1:172.21.15.116:6789/0 conn(0x7fc018070d90 legacy=0x7fc0180711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01d266700 1 -- 172.21.15.116:0/1569378590 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc018071ac0 con 0x7fc01815e0b0 2023-11-08T08:21:42.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc0177fe700 1 -- 172.21.15.116:0/1569378590 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655106413 0 0) 0x7fc018151a50 con 0x7fc018070d90 2023-11-08T08:21:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01ca65700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc01d266700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01815e0b0 0x7fc01815e530 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc010009d90 tx=0x7fc01000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3baa445ae03313ce server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.918+0000 7fc0177fe700 1 -- 172.21.15.116:0/1569378590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000e070 con 0x7fc01815e0b0 2023-11-08T08:21:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc0177fe700 1 -- 172.21.15.116:0/1569378590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01000bbd0 con 0x7fc01815e0b0 2023-11-08T08:21:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc0177fe700 1 -- 172.21.15.116:0/1569378590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010012430 con 0x7fc01815e0b0 2023-11-08T08:21:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1569378590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01815e0b0 msgr2=0x7fc01815e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01815e0b0 0x7fc01815e530 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc010009d90 tx=0x7fc01000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1569378590 shutdown_connections 2023-11-08T08:21:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc01815e0b0 0x7fc01815e530 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1569378590 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1569378590 >> 172.21.15.116:0/1569378590 conn(0x7fc01806c010 msgr2=0x7fc0180cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1569378590 shutdown_connections 2023-11-08T08:21:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.919+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1569378590 wait complete. 2023-11-08T08:21:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.920+0000 7fc01ecc9700 1 Processor -- start 2023-11-08T08:21:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.920+0000 7fc01ecc9700 1 -- start start 2023-11-08T08:21:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc01ecc9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01814f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc01ecc9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 0x7fc0181f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc01ecc9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01815e0b0 0x7fc0181f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc01ecc9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181f1400 con 0x7fc018070d90 2023-11-08T08:21:42.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc01ecc9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181f2240 con 0x7fc01815e0b0 2023-11-08T08:21:42.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc01ecc9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181f1630 con 0x7fc01806a270 2023-11-08T08:21:42.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc017fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 0x7fc0181f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc017fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 0x7fc0181f0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45902/0 (socket says 172.21.15.116:45902) 2023-11-08T08:21:42.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.921+0000 7fc017fff700 1 -- 172.21.15.116:0/1881455494 learned_addr learned my addr 172.21.15.116:0/1881455494 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:42.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc01d266700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01815e0b0 0x7fc0181f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc01ca65700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01814f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc017fff700 1 -- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 msgr2=0x7fc01814f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc017fff700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01814f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc017fff700 1 -- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01815e0b0 msgr2=0x7fc0181f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc017fff700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01815e0b0 0x7fc0181f0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc017fff700 1 -- 172.21.15.116:0/1881455494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010009a70 con 0x7fc018070d90 2023-11-08T08:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc01d266700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01815e0b0 0x7fc0181f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc017fff700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 0x7fc0181f0830 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc0040011e0 tx=0x7fc00400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004004c80 con 0x7fc018070d90 2023-11-08T08:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004004e00 con 0x7fc018070d90 2023-11-08T08:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01815b0d0 con 0x7fc018070d90 2023-11-08T08:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004007310 con 0x7fc018070d90 2023-11-08T08:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.922+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01815adf0 con 0x7fc018070d90 2023-11-08T08:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.923+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01815aac0 con 0x7fc018070d90 2023-11-08T08:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.923+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc00401a020 con 0x7fc018070d90 2023-11-08T08:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.924+0000 7fc0157fa700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbffc0682f0 0x7fbffc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.924+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc004085910 con 0x7fc018070d90 2023-11-08T08:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.924+0000 7fc01ca65700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbffc0682f0 0x7fbffc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.924+0000 7fc01ca65700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbffc0682f0 0x7fbffc06a7a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc00c006fd0 tx=0x7fc00c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:42.928+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc004052170 con 0x7fc018070d90 2023-11-08T08:21:43.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.108+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0181f23b0 con 0x7fc018070d90 2023-11-08T08:21:43.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.109+0000 7fc0157fa700 1 -- 172.21.15.116:0/1881455494 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17959 (secure 0 0 0) 0x7fc004055790 con 0x7fc018070d90 2023-11-08T08:21:43.110 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:43.111 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:33.084884+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbffc0682f0 msgr2=0x7fbffc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbffc0682f0 0x7fbffc06a7a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc00c006fd0 tx=0x7fc00c008040 comp rx=0 tx=0).stop 2023-11-08T08:21:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 msgr2=0x7fc0181f0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 0x7fc0181f0830 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc0040011e0 tx=0x7fc00400cae0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 shutdown_connections 2023-11-08T08:21:43.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbffc0682f0 0x7fbffc06a7a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc01815e0b0 0x7fc0181f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc01806a270 0x7fc01814f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 --2- 172.21.15.116:0/1881455494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc018070d90 0x7fc0181f0830 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 >> 172.21.15.116:0/1881455494 conn(0x7fc01806c010 msgr2=0x7fc01815d150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:43.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 shutdown_connections 2023-11-08T08:21:43.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.111+0000 7fc01ecc9700 1 -- 172.21.15.116:0/1881455494 wait complete. 2023-11-08T08:21:43.124 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:21:29.779880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:21:31.046244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:21:43.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:21:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.246+0000 7f5cafe11700 1 Processor -- start 2023-11-08T08:21:43.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.247+0000 7f5cafe11700 1 -- start start 2023-11-08T08:21:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.247+0000 7f5cafe11700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca80cf710 0x7f5ca80ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.247+0000 7f5cafe11700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ca8071890 0x7f5ca8071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.247+0000 7f5cafe11700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca815cc10 con 0x7f5ca806a270 2023-11-08T08:21:43.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.247+0000 7f5cafe11700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca806c9b0 con 0x7f5ca8071890 2023-11-08T08:21:43.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.247+0000 7f5cafe11700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca806cc90 con 0x7f5ca80cf710 2023-11-08T08:21:43.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ca8071890 0x7f5ca8071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ca8071890 0x7f5ca8071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45910/0 (socket says 172.21.15.116:45910) 2023-11-08T08:21:43.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 -- 172.21.15.116:0/840058183 learned_addr learned my addr 172.21.15.116:0/840058183 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:43.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cadbad700 1 --2- 172.21.15.116:0/840058183 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca80cf710 0x7f5ca80ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 -- 172.21.15.116:0/840058183 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca80cf710 msgr2=0x7f5ca80ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:43.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 --2- 172.21.15.116:0/840058183 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca80cf710 0x7f5ca80ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 -- 172.21.15.116:0/840058183 >> v1:172.21.15.129:6789/0 conn(0x7f5ca806a270 legacy=0x7f5ca80b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 -- 172.21.15.116:0/840058183 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca806cf70 con 0x7f5ca8071890 2023-11-08T08:21:43.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cad3ac700 1 --2- 172.21.15.116:0/840058183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca8071890 0x7f5ca8071220 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40098f0 tx=0x7f5ca4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e91f916e8c509f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cacbab700 1 -- 172.21.15.116:0/840058183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400b8a0 con 0x7f5ca8071890 2023-11-08T08:21:43.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cacbab700 1 -- 172.21.15.116:0/840058183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca400ba00 con 0x7f5ca8071890 2023-11-08T08:21:43.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.248+0000 7f5cacbab700 1 -- 172.21.15.116:0/840058183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400bd90 con 0x7f5ca8071890 2023-11-08T08:21:43.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 -- 172.21.15.116:0/840058183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca8071890 msgr2=0x7f5ca8071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 --2- 172.21.15.116:0/840058183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca8071890 0x7f5ca8071220 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40098f0 tx=0x7f5ca4009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:43.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 -- 172.21.15.116:0/840058183 shutdown_connections 2023-11-08T08:21:43.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 --2- 172.21.15.116:0/840058183 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca80cf710 0x7f5ca80ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 --2- 172.21.15.116:0/840058183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca8071890 0x7f5ca8071220 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 -- 172.21.15.116:0/840058183 >> 172.21.15.116:0/840058183 conn(0x7f5ca806c010 msgr2=0x7f5ca80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:43.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 -- 172.21.15.116:0/840058183 shutdown_connections 2023-11-08T08:21:43.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.249+0000 7f5cafe11700 1 -- 172.21.15.116:0/840058183 wait complete. 2023-11-08T08:21:43.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.250+0000 7f5cafe11700 1 Processor -- start 2023-11-08T08:21:43.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.250+0000 7f5cafe11700 1 -- start start 2023-11-08T08:21:43.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cafe11700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca806a270 0x7f5ca80cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cafe11700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ca8071890 0x7f5ca814dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cafe11700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 0x7f5ca80b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cafe11700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80cc680 con 0x7f5ca80cf710 2023-11-08T08:21:43.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cafe11700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca81f1f10 con 0x7f5ca8071890 2023-11-08T08:21:43.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cafe11700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca814e120 con 0x7f5ca806a270 2023-11-08T08:21:43.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cae3ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 0x7f5ca80b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cae3ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 0x7f5ca80b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45912/0 (socket says 172.21.15.116:45912) 2023-11-08T08:21:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.251+0000 7f5cae3ae700 1 -- 172.21.15.116:0/1110417711 learned_addr learned my addr 172.21.15.116:0/1110417711 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cae3ae700 1 -- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca806a270 msgr2=0x7f5ca80cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cae3ae700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca806a270 0x7f5ca80cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cae3ae700 1 -- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ca8071890 msgr2=0x7f5ca814dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cad3ac700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ca8071890 0x7f5ca814dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cae3ae700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ca8071890 0x7f5ca814dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cae3ae700 1 -- 172.21.15.116:0/1110417711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4009550 con 0x7f5ca80cf710 2023-11-08T08:21:43.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cae3ae700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 0x7f5ca80b1a70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000cc50 tx=0x7f5ca000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca000ad80 con 0x7f5ca80cf710 2023-11-08T08:21:43.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca000ce50 con 0x7f5ca80cf710 2023-11-08T08:21:43.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0003c40 con 0x7f5ca80cf710 2023-11-08T08:21:43.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca806d530 con 0x7f5ca80cf710 2023-11-08T08:21:43.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.252+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca806d250 con 0x7f5ca80cf710 2023-11-08T08:21:43.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.253+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca806a6f0 con 0x7f5ca80cf710 2023-11-08T08:21:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.253+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5ca0010050 con 0x7f5ca80cf710 2023-11-08T08:21:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.254+0000 7f5c9e7fc700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5c8c0682f0 0x7f5c8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.254+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5ca0085d10 con 0x7f5ca80cf710 2023-11-08T08:21:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.254+0000 7f5cadbad700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5c8c0682f0 0x7f5c8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.254+0000 7f5cadbad700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5c8c0682f0 0x7f5c8c06a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5c980098d0 tx=0x7f5c98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.259+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5ca008b020 con 0x7f5ca80cf710 2023-11-08T08:21:43.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.434+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5ca80cc7c0 con 0x7f5ca80cf710 2023-11-08T08:21:43.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.435+0000 7f5c9e7fc700 1 -- 172.21.15.116:0/1110417711 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5ca80cc7c0 con 0x7f5ca80cf710 2023-11-08T08:21:43.436 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:21:43.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5c8c0682f0 msgr2=0x7f5c8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5c8c0682f0 0x7f5c8c06a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5c980098d0 tx=0x7f5c98008040 comp rx=0 tx=0).stop 2023-11-08T08:21:43.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 msgr2=0x7f5ca80b1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 0x7f5ca80b1a70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000cc50 tx=0x7f5ca000a340 comp rx=0 tx=0).stop 2023-11-08T08:21:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 shutdown_connections 2023-11-08T08:21:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ca80cf710 0x7f5ca80b1a70 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5ca806a270 0x7f5ca80cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f5c8c0682f0 0x7f5c8c06a7a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 --2- 172.21.15.116:0/1110417711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ca8071890 0x7f5ca814dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 >> 172.21.15.116:0/1110417711 conn(0x7f5ca806c010 msgr2=0x7f5ca80b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 shutdown_connections 2023-11-08T08:21:43.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.437+0000 7f5cafe11700 1 -- 172.21.15.116:0/1110417711 wait complete. 2023-11-08T08:21:43.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:21:43.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.572+0000 7f339a3c1700 1 Processor -- start 2023-11-08T08:21:43.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.572+0000 7f339a3c1700 1 -- start start 2023-11-08T08:21:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.573+0000 7f339a3c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f33940b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.573+0000 7f339a3c1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3394159ec0 con 0x7f33941599b0 2023-11-08T08:21:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.573+0000 7f339a3c1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339415aa60 con 0x7f33940af0a0 2023-11-08T08:21:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.573+0000 7f339a3c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33940b1f60 con 0x7f33940b1650 2023-11-08T08:21:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f33937fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f33941599b0 0x7f339415d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39438/0 (socket says 172.21.15.116:39438) 2023-11-08T08:21:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f33937fe700 1 -- 172.21.15.116:0/204834998 learned_addr learned my addr 172.21.15.116:0/204834998 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f33940b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691457223 0 0) 0x7f3394159ec0 con 0x7f33941599b0 2023-11-08T08:21:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33800035b0 con 0x7f33941599b0 2023-11-08T08:21:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227758557 0 0) 0x7f33800035b0 con 0x7f33941599b0 2023-11-08T08:21:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 msgr2=0x7f33940b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f33940b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 >> v1:172.21.15.129:6789/0 conn(0x7f33940af0a0 legacy=0x7f339415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339415d8a0 con 0x7f33941599b0 2023-11-08T08:21:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f337c001670 con 0x7f33941599b0 2023-11-08T08:21:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 >> v1:172.21.15.116:6789/0 conn(0x7f33941599b0 legacy=0x7f339415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3380003a30 0x7f3380003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f3380004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380004d90 0x7f3380005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33940b1f60 con 0x7f3380003a30 2023-11-08T08:21:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338000bac0 con 0x7f3380004d90 2023-11-08T08:21:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338000bc50 con 0x7f33940b1650 2023-11-08T08:21:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3380003a30 0x7f3380003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f33937fe700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f3380004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f339895e700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380004d90 0x7f3380005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 -- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 msgr2=0x7f3380004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:43.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f3380004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 -- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380004d90 msgr2=0x7f3380005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:43.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380004d90 0x7f3380005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f3393fff700 1 -- 172.21.15.116:0/204834998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800035b0 con 0x7f3380003a30 2023-11-08T08:21:43.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.574+0000 7f33937fe700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940b1650 0x7f3380004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:43.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f3393fff700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3380003a30 0x7f3380003e40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f33840096f0 tx=0x7f3380005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f210c7168ebdda70 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338400e6c0 con 0x7f3380003a30 2023-11-08T08:21:43.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338400e820 con 0x7f3380003a30 2023-11-08T08:21:43.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f3392ffd700 1 -- 172.21.15.116:0/204834998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338400ebb0 con 0x7f3380003a30 2023-11-08T08:21:43.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339a3c1700 1 -- 172.21.15.116:0/204834998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3380003a30 msgr2=0x7f3380003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339a3c1700 1 --2- 172.21.15.116:0/204834998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3380003a30 0x7f3380003e40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f33840096f0 tx=0x7f3380005880 comp rx=0 tx=0).stop 2023-11-08T08:21:43.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339895e700 1 -- 172.21.15.116:0/204834998 reap_dead start 2023-11-08T08:21:43.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339a3c1700 1 -- 172.21.15.116:0/204834998 shutdown_connections 2023-11-08T08:21:43.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339a3c1700 1 -- 172.21.15.116:0/204834998 >> 172.21.15.116:0/204834998 conn(0x7f339406c010 msgr2=0x7f33940cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:43.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339a3c1700 1 -- 172.21.15.116:0/204834998 shutdown_connections 2023-11-08T08:21:43.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.575+0000 7f339a3c1700 1 -- 172.21.15.116:0/204834998 wait complete. 2023-11-08T08:21:43.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.576+0000 7f339a3c1700 1 Processor -- start 2023-11-08T08:21:43.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 -- start start 2023-11-08T08:21:43.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380003a30 0x7f33941f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 0x7f3394156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940d0210 0x7f3394156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394144e50 con 0x7f33940cfda0 2023-11-08T08:21:43.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941450f0 con 0x7f3380003a30 2023-11-08T08:21:43.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f339a3c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33941f62d0 con 0x7f33940d0210 2023-11-08T08:21:43.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f33937fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 0x7f3394156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.577+0000 7f33937fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 0x7f3394156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45944/0 (socket says 172.21.15.116:45944) 2023-11-08T08:21:43.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 -- 172.21.15.116:0/1943480576 learned_addr learned my addr 172.21.15.116:0/1943480576 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:43.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f3393fff700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380003a30 0x7f33941f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f339895e700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940d0210 0x7f3394156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 -- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940d0210 msgr2=0x7f3394156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940d0210 0x7f3394156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 -- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380003a30 msgr2=0x7f33941f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380003a30 0x7f33941f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 -- 172.21.15.116:0/1943480576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339415de60 con 0x7f33940cfda0 2023-11-08T08:21:43.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f33937fe700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 0x7f3394156950 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f337c00e8f0 tx=0x7f337c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00c620 con 0x7f33940cfda0 2023-11-08T08:21:43.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c00c780 con 0x7f33940cfda0 2023-11-08T08:21:43.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339415db80 con 0x7f33940cfda0 2023-11-08T08:21:43.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00cb10 con 0x7f33940cfda0 2023-11-08T08:21:43.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.578+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33940b1a20 con 0x7f33940cfda0 2023-11-08T08:21:43.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.579+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394145260 con 0x7f33940cfda0 2023-11-08T08:21:43.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.579+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f337c00cc70 con 0x7f33940cfda0 2023-11-08T08:21:43.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.580+0000 7f3390ff9700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33740682f0 0x7f337406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.580+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f337c085d90 con 0x7f33940cfda0 2023-11-08T08:21:43.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.580+0000 7f3393fff700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33740682f0 0x7f337406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.580+0000 7f3393fff700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33740682f0 0x7f337406a7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f338401c040 tx=0x7f3384011040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.585+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f337c052500 con 0x7f33940cfda0 2023-11-08T08:21:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.764+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33940743f0 con 0x7f33940cfda0 2023-11-08T08:21:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.764+0000 7f3390ff9700 1 -- 172.21.15.116:0/1943480576 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f337c055b20 con 0x7f33940cfda0 2023-11-08T08:21:43.766 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:21:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33740682f0 msgr2=0x7f337406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33740682f0 0x7f337406a7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f338401c040 tx=0x7f3384011040 comp rx=0 tx=0).stop 2023-11-08T08:21:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 msgr2=0x7f3394156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 0x7f3394156950 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f337c00e8f0 tx=0x7f337c00ec00 comp rx=0 tx=0).stop 2023-11-08T08:21:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 shutdown_connections 2023-11-08T08:21:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f33940d0210 0x7f3394156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f33740682f0 0x7f337406a7a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3380003a30 0x7f33941f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 --2- 172.21.15.116:0/1943480576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33940cfda0 0x7f3394156950 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 >> 172.21.15.116:0/1943480576 conn(0x7f339406c010 msgr2=0x7f339415cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 shutdown_connections 2023-11-08T08:21:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.766+0000 7f339a3c1700 1 -- 172.21.15.116:0/1943480576 wait complete. 2023-11-08T08:21:43.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:21:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.901+0000 7fb7d9865700 1 Processor -- start 2023-11-08T08:21:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.901+0000 7fb7d9865700 1 -- start start 2023-11-08T08:21:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d9865700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb7d40b1650 0x7fb7d40b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d9865700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7d41599b0 0x7fb7d415d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d9865700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d4159ec0 con 0x7fb7d40af0a0 2023-11-08T08:21:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d9865700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d415aa60 con 0x7fb7d41599b0 2023-11-08T08:21:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d9865700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d40b1f60 con 0x7fb7d40b1650 2023-11-08T08:21:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d27fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7d41599b0 0x7fb7d415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d37fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb7d40af0a0 0x7fb7d415a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39454/0 (socket says 172.21.15.116:39454) 2023-11-08T08:21:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d37fe700 1 -- 172.21.15.116:0/704263576 learned_addr learned my addr 172.21.15.116:0/704263576 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d2ffd700 1 --2- 172.21.15.116:0/704263576 >> v2:172.21.15.129:3300/0 conn(0x7fb7d40b1650 0x7fb7d40b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d27fc700 1 -- 172.21.15.116:0/704263576 >> v2:172.21.15.129:3300/0 conn(0x7fb7d40b1650 msgr2=0x7fb7d40b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d27fc700 1 --2- 172.21.15.116:0/704263576 >> v2:172.21.15.129:3300/0 conn(0x7fb7d40b1650 0x7fb7d40b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d27fc700 1 -- 172.21.15.116:0/704263576 >> v1:172.21.15.116:6789/0 conn(0x7fb7d40af0a0 legacy=0x7fb7d415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d1ffb700 1 -- 172.21.15.116:0/704263576 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740425246 0 0) 0x7fb7d4159ec0 con 0x7fb7d40af0a0 2023-11-08T08:21:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d27fc700 1 -- 172.21.15.116:0/704263576 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d415d8a0 con 0x7fb7d41599b0 2023-11-08T08:21:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d27fc700 1 --2- 172.21.15.116:0/704263576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d41599b0 0x7fb7d415d050 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4067310 tx=0x7fb7c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d78f067c7fcd113 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d1ffb700 1 -- 172.21.15.116:0/704263576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4011830 con 0x7fb7d41599b0 2023-11-08T08:21:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d1ffb700 1 -- 172.21.15.116:0/704263576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c4011990 con 0x7fb7d41599b0 2023-11-08T08:21:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.902+0000 7fb7d1ffb700 1 -- 172.21.15.116:0/704263576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4011d20 con 0x7fb7d41599b0 2023-11-08T08:21:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 -- 172.21.15.116:0/704263576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d41599b0 msgr2=0x7fb7d415d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 --2- 172.21.15.116:0/704263576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d41599b0 0x7fb7d415d050 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4067310 tx=0x7fb7c4009b20 comp rx=0 tx=0).stop 2023-11-08T08:21:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 -- 172.21.15.116:0/704263576 shutdown_connections 2023-11-08T08:21:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 --2- 172.21.15.116:0/704263576 >> v2:172.21.15.129:3300/0 conn(0x7fb7d40b1650 0x7fb7d40b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 --2- 172.21.15.116:0/704263576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d41599b0 0x7fb7d415d050 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 -- 172.21.15.116:0/704263576 >> 172.21.15.116:0/704263576 conn(0x7fb7d406c010 msgr2=0x7fb7d40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 -- 172.21.15.116:0/704263576 shutdown_connections 2023-11-08T08:21:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.903+0000 7fb7d9865700 1 -- 172.21.15.116:0/704263576 wait complete. 2023-11-08T08:21:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.904+0000 7fb7d9865700 1 Processor -- start 2023-11-08T08:21:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.904+0000 7fb7d9865700 1 -- start start 2023-11-08T08:21:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d9865700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d40af0a0 0x7fb7d41e4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d9865700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 0x7fb7d414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d9865700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7d415c3d0 0x7fb7d40c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d9865700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41e45b0 con 0x7fb7d40b1650 2023-11-08T08:21:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d9865700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d415b330 con 0x7fb7d40af0a0 2023-11-08T08:21:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d9865700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d415b610 con 0x7fb7d415c3d0 2023-11-08T08:21:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 0x7fb7d414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 0x7fb7d414df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45952/0 (socket says 172.21.15.116:45952) 2023-11-08T08:21:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 -- 172.21.15.116:0/3693831518 learned_addr learned my addr 172.21.15.116:0/3693831518 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d37fe700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7d415c3d0 0x7fb7d40c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d2ffd700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d40af0a0 0x7fb7d41e4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 -- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7d415c3d0 msgr2=0x7fb7d40c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7d415c3d0 0x7fb7d40c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 -- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d40af0a0 msgr2=0x7fb7d41e4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d40af0a0 0x7fb7d41e4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.905+0000 7fb7d27fc700 1 -- 172.21.15.116:0/3693831518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c40094a0 con 0x7fb7d40b1650 2023-11-08T08:21:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.906+0000 7fb7d27fc700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 0x7fb7d414df00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4014040 tx=0x7fb7c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.906+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40038a0 con 0x7fb7d40b1650 2023-11-08T08:21:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.906+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c401e070 con 0x7fb7d40b1650 2023-11-08T08:21:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.906+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4003c20 con 0x7fb7d40b1650 2023-11-08T08:21:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.906+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d415de60 con 0x7fb7d40b1650 2023-11-08T08:21:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.906+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d415db80 con 0x7fb7d40b1650 2023-11-08T08:21:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.907+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d415d590 con 0x7fb7d40b1650 2023-11-08T08:21:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.907+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7c400f050 con 0x7fb7d40b1650 2023-11-08T08:21:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.907+0000 7fb7bbfff700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7c00682f0 0x7fb7c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.908+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb7c4086c90 con 0x7fb7d40b1650 2023-11-08T08:21:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.908+0000 7fb7d2ffd700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7c00682f0 0x7fb7c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.908+0000 7fb7d2ffd700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7c00682f0 0x7fb7c006a7a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009930 tx=0x7fb7c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:43.912+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7c40534f0 con 0x7fb7d40b1650 2023-11-08T08:21:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.089+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7d40743f0 con 0x7fb7d40b1650 2023-11-08T08:21:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.090+0000 7fb7bbfff700 1 -- 172.21.15.116:0/3693831518 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7c4015020 con 0x7fb7d40b1650 2023-11-08T08:21:44.091 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:21:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7c00682f0 msgr2=0x7fb7c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7c00682f0 0x7fb7c006a7a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009930 tx=0x7fb7c8008040 comp rx=0 tx=0).stop 2023-11-08T08:21:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 msgr2=0x7fb7d414df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 0x7fb7d414df00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4014040 tx=0x7fb7c4004750 comp rx=0 tx=0).stop 2023-11-08T08:21:44.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 shutdown_connections 2023-11-08T08:21:44.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7d415c3d0 0x7fb7d40c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb7c00682f0 0x7fb7c006a7a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d40af0a0 0x7fb7d41e4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 --2- 172.21.15.116:0/3693831518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d40b1650 0x7fb7d414df00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 >> 172.21.15.116:0/3693831518 conn(0x7fb7d406c010 msgr2=0x7fb7d40af630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:44.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 shutdown_connections 2023-11-08T08:21:44.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.092+0000 7fb7d9865700 1 -- 172.21.15.116:0/3693831518 wait complete. 2023-11-08T08:21:44.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:21:44.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.226+0000 7f48048ee700 1 Processor -- start 2023-11-08T08:21:44.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.226+0000 7f48048ee700 1 -- start start 2023-11-08T08:21:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f48048ee700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 0x7f48000b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f48048ee700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800159ec0 con 0x7f48000af0a0 2023-11-08T08:21:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f48048ee700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480015aa60 con 0x7f48001599b0 2023-11-08T08:21:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f48048ee700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b1f60 con 0x7f48000b1650 2023-11-08T08:21:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f47fed9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f48000af0a0 0x7f480015a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39458/0 (socket says 172.21.15.116:39458) 2023-11-08T08:21:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f47fed9d700 1 -- 172.21.15.116:0/3094587342 learned_addr learned my addr 172.21.15.116:0/3094587342 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f47fe59c700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 0x7f48000b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918408671 0 0) 0x7f4800159ec0 con 0x7f48000af0a0 2023-11-08T08:21:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.228+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e80035b0 con 0x7f48000af0a0 2023-11-08T08:21:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695890812 0 0) 0x7f480015aa60 con 0x7f48001599b0 2023-11-08T08:21:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800159ec0 con 0x7f48001599b0 2023-11-08T08:21:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238505574 0 0) 0x7f47e80035b0 con 0x7f48000af0a0 2023-11-08T08:21:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 msgr2=0x7f48000b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 0x7f48000b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 >> v1:172.21.15.129:6789/0 conn(0x7f48001599b0 legacy=0x7f480015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480015d8a0 con 0x7f48000af0a0 2023-11-08T08:21:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f47f80019f0 con 0x7f48000af0a0 2023-11-08T08:21:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 >> v1:172.21.15.116:6789/0 conn(0x7f48000af0a0 legacy=0x7f480015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8003bc0 0x7f47e8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 0x7f47e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b1f60 con 0x7f47e8004d90 2023-11-08T08:21:44.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e800bac0 con 0x7f47e8003bc0 2023-11-08T08:21:44.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e800bc30 con 0x7f48000b1650 2023-11-08T08:21:44.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fe59c700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8003bc0 0x7f47e8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fdd9b700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 0x7f47e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 -- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 msgr2=0x7f47e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:44.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000b1650 0x7f47e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 -- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8003bc0 msgr2=0x7f47e8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:44.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8003bc0 0x7f47e8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 -- 172.21.15.116:0/3094587342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e80035b0 con 0x7f47e8004d90 2023-11-08T08:21:44.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f47f800a7a0 tx=0x7f47e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287acd9c8ab7318f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f80072d0 con 0x7f47e8004d90 2023-11-08T08:21:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8007450 con 0x7f47e8004d90 2023-11-08T08:21:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fd59a700 1 -- 172.21.15.116:0/3094587342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f8007800 con 0x7f47e8004d90 2023-11-08T08:21:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f48048ee700 1 -- 172.21.15.116:0/3094587342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e8004d90 msgr2=0x7f47e8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f48048ee700 1 --2- 172.21.15.116:0/3094587342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f47f800a7a0 tx=0x7f47e8005880 comp rx=0 tx=0).stop 2023-11-08T08:21:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f47fed9d700 1 -- 172.21.15.116:0/3094587342 reap_dead start 2023-11-08T08:21:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f48048ee700 1 -- 172.21.15.116:0/3094587342 shutdown_connections 2023-11-08T08:21:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f48048ee700 1 -- 172.21.15.116:0/3094587342 >> 172.21.15.116:0/3094587342 conn(0x7f480006c010 msgr2=0x7f48000cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f48048ee700 1 -- 172.21.15.116:0/3094587342 shutdown_connections 2023-11-08T08:21:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.229+0000 7f48048ee700 1 -- 172.21.15.116:0/3094587342 wait complete. 2023-11-08T08:21:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.230+0000 7f48048ee700 1 Processor -- start 2023-11-08T08:21:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 -- start start 2023-11-08T08:21:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8004d90 0x7f480006a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 0x7f4800158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000c5d10 0x7f48000c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48001f3660 con 0x7f4800158a20 2023-11-08T08:21:44.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48001f2620 con 0x7f47e8004d90 2023-11-08T08:21:44.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48001f28a0 con 0x7f48000c5d10 2023-11-08T08:21:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 0x7f4800158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 0x7f4800158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45968/0 (socket says 172.21.15.116:45968) 2023-11-08T08:21:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 -- 172.21.15.116:0/598077747 learned_addr learned my addr 172.21.15.116:0/598077747 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 -- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000c5d10 msgr2=0x7f48000c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000c5d10 0x7f48000c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 -- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8004d90 msgr2=0x7f480006a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fe59c700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8004d90 0x7f480006a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8004d90 0x7f480006a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 -- 172.21.15.116:0/598077747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480015de60 con 0x7f4800158a20 2023-11-08T08:21:44.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fe59c700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8004d90 0x7f480006a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:44.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47fdd9b700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 0x7f4800158e90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f47f000f770 tx=0x7f47f000fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:44.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f00052e0 con 0x7f4800158a20 2023-11-08T08:21:44.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f0005440 con 0x7f4800158a20 2023-11-08T08:21:44.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480015db80 con 0x7f4800158a20 2023-11-08T08:21:44.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f00038c0 con 0x7f4800158a20 2023-11-08T08:21:44.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.232+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48000b1a20 con 0x7f4800158a20 2023-11-08T08:21:44.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.234+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48000743f0 con 0x7f4800158a20 2023-11-08T08:21:44.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.234+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47f0003a20 con 0x7f4800158a20 2023-11-08T08:21:44.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.234+0000 7f47eeffd700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47e00682f0 0x7f47e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.234+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f47f0085c60 con 0x7f4800158a20 2023-11-08T08:21:44.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.234+0000 7f47fe59c700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47e00682f0 0x7f47e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.234+0000 7f47fe59c700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47e00682f0 0x7f47e006a7a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009b50 tx=0x7f47f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:44.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.239+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47f00524c0 con 0x7f4800158a20 2023-11-08T08:21:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.419+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f480006e620 con 0x7f4800158a20 2023-11-08T08:21:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.419+0000 7f47eeffd700 1 -- 172.21.15.116:0/598077747 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47f0055ae0 con 0x7f4800158a20 2023-11-08T08:21:44.420 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:21:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47e00682f0 msgr2=0x7f47e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47e00682f0 0x7f47e006a7a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009b50 tx=0x7f47f4008040 comp rx=0 tx=0).stop 2023-11-08T08:21:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 msgr2=0x7f4800158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 0x7f4800158e90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f47f000f770 tx=0x7f47f000fb30 comp rx=0 tx=0).stop 2023-11-08T08:21:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 shutdown_connections 2023-11-08T08:21:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f48000c5d10 0x7f48000c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f47e00682f0 0x7f47e006a7a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47e8004d90 0x7f480006a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 --2- 172.21.15.116:0/598077747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4800158a20 0x7f4800158e90 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 >> 172.21.15.116:0/598077747 conn(0x7f480006c010 msgr2=0x7f480015a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 shutdown_connections 2023-11-08T08:21:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.421+0000 7f48048ee700 1 -- 172.21.15.116:0/598077747 wait complete. 2023-11-08T08:21:44.433 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-08T08:21:44.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-08T08:21:44.504+0000 7f50f4469700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-11-08T08:21:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.556+0000 7fc903109700 1 Processor -- start 2023-11-08T08:21:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.556+0000 7fc903109700 1 -- start start 2023-11-08T08:21:44.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc903109700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 0x7fc8fc0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc903109700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc15a9c0 con 0x7fc8fc0cc200 2023-11-08T08:21:44.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc903109700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc0c7b80 con 0x7fc8fc0b2ee0 2023-11-08T08:21:44.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc903109700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0cc990 con 0x7fc8fc15a5f0 2023-11-08T08:21:44.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc8fbfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8fc0cc200 0x7fc8fc0b27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39474/0 (socket says 172.21.15.116:39474) 2023-11-08T08:21:44.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc8fbfff700 1 -- 172.21.15.116:0/3090608502 learned_addr learned my addr 172.21.15.116:0/3090608502 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:44.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231846505 0 0) 0x7fc8fc15a9c0 con 0x7fc8fc0cc200 2023-11-08T08:21:44.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec0035b0 con 0x7fc8fc0cc200 2023-11-08T08:21:44.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.557+0000 7fc900ea5700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 0x7fc8fc0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600575936 0 0) 0x7fc8ec0035b0 con 0x7fc8fc0cc200 2023-11-08T08:21:44.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 msgr2=0x7fc8fc0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 0x7fc8fc0d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 >> v1:172.21.15.129:6789/0 conn(0x7fc8fc0b2ee0 legacy=0x7fc8fc0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc0ccc70 con 0x7fc8fc0cc200 2023-11-08T08:21:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fc8e8001670 con 0x7fc8fc0cc200 2023-11-08T08:21:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 >> v1:172.21.15.116:6789/0 conn(0x7fc8fc0cc200 legacy=0x7fc8fc0b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 0x7fc8ec003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8ec003fb0 0x7fc8ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8ec004d90 0x7fc8ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0cc990 con 0x7fc8ec004d90 2023-11-08T08:21:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec00baf0 con 0x7fc8ec003fb0 2023-11-08T08:21:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec00bcb0 con 0x7fc8fc15a5f0 2023-11-08T08:21:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8ec004d90 0x7fc8ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fbfff700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8ec003fb0 0x7fc8ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc900ea5700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 0x7fc8ec003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 -- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 msgr2=0x7fc8ec003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc15a5f0 0x7fc8ec003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 -- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8ec003fb0 msgr2=0x7fc8ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8ec003fb0 0x7fc8ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 -- 172.21.15.116:0/3090608502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0035b0 con 0x7fc8ec004d90 2023-11-08T08:21:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8ec004d90 0x7fc8ec005200 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400c210 tx=0x7fc8ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b89fa6f510a331a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4001df0 con 0x7fc8ec004d90 2023-11-08T08:21:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f400cd70 con 0x7fc8ec004d90 2023-11-08T08:21:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.558+0000 7fc8fb7fe700 1 -- 172.21.15.116:0/3090608502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003c40 con 0x7fc8ec004d90 2023-11-08T08:21:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc903109700 1 -- 172.21.15.116:0/3090608502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8ec004d90 msgr2=0x7fc8ec005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc903109700 1 --2- 172.21.15.116:0/3090608502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8ec004d90 0x7fc8ec005200 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400c210 tx=0x7fc8ec005880 comp rx=0 tx=0).stop 2023-11-08T08:21:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc9016a6700 1 -- 172.21.15.116:0/3090608502 reap_dead start 2023-11-08T08:21:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc903109700 1 -- 172.21.15.116:0/3090608502 shutdown_connections 2023-11-08T08:21:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc903109700 1 -- 172.21.15.116:0/3090608502 >> 172.21.15.116:0/3090608502 conn(0x7fc8fc06ea80 msgr2=0x7fc8fc06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc903109700 1 -- 172.21.15.116:0/3090608502 shutdown_connections 2023-11-08T08:21:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.559+0000 7fc903109700 1 -- 172.21.15.116:0/3090608502 wait complete. 2023-11-08T08:21:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.560+0000 7fc903109700 1 Processor -- start 2023-11-08T08:21:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.560+0000 7fc903109700 1 -- start start 2023-11-08T08:21:44.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.561+0000 7fc903109700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8fc14b040 0x7fc8fc0a84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.561+0000 7fc903109700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc14b410 0x7fc8fc1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.561+0000 7fc903109700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 0x7fc8fc15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc903109700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc157c50 con 0x7fc8fc1577e0 2023-11-08T08:21:44.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc903109700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0a8a20 con 0x7fc8fc14b040 2023-11-08T08:21:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc903109700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc1f4890 con 0x7fc8fc14b410 2023-11-08T08:21:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 0x7fc8fc15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 0x7fc8fc15a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45986/0 (socket says 172.21.15.116:45986) 2023-11-08T08:21:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 -- 172.21.15.116:0/3750636743 learned_addr learned my addr 172.21.15.116:0/3750636743 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc900ea5700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8fc14b040 0x7fc8fc0a84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc8fbfff700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc14b410 0x7fc8fc1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 -- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc14b410 msgr2=0x7fc8fc1572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc14b410 0x7fc8fc1572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 -- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8fc14b040 msgr2=0x7fc8fc0a84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8fc14b040 0x7fc8fc0a84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 -- 172.21.15.116:0/3750636743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc0cd230 con 0x7fc8fc1577e0 2023-11-08T08:21:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc8fbfff700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc14b410 0x7fc8fc1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc9016a6700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 0x7fc8fc15a9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4015040 tx=0x7fc8f40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003e10 con 0x7fc8fc1577e0 2023-11-08T08:21:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f40050d0 con 0x7fc8fc1577e0 2023-11-08T08:21:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4005480 con 0x7fc8fc1577e0 2023-11-08T08:21:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc0ccf50 con 0x7fc8fc1577e0 2023-11-08T08:21:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.562+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc0cd510 con 0x7fc8fc1577e0 2023-11-08T08:21:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.563+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc15e770 con 0x7fc8fc1577e0 2023-11-08T08:21:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.563+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8f4010050 con 0x7fc8fc1577e0 2023-11-08T08:21:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.564+0000 7fc8f97fa700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8e00682f0 0x7fc8e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.564+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc8f4086450 con 0x7fc8fc1577e0 2023-11-08T08:21:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.564+0000 7fc900ea5700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8e00682f0 0x7fc8e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.564+0000 7fc900ea5700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8e00682f0 0x7fc8e006a7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009c60 tx=0x7fc8f0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.568+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8fc15e770 con 0x7fc8fc1577e0 2023-11-08T08:21:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.763+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc8fc15e770 con 0x7fc8fc1577e0 2023-11-08T08:21:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.764+0000 7fc8f97fa700 1 -- 172.21.15.116:0/3750636743 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc8fc15e770 con 0x7fc8fc1577e0 2023-11-08T08:21:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.765+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8e00682f0 msgr2=0x7fc8e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.765+0000 7fc903109700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8e00682f0 0x7fc8e006a7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009c60 tx=0x7fc8f0009210 comp rx=0 tx=0).stop 2023-11-08T08:21:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.765+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 msgr2=0x7fc8fc15a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.765+0000 7fc903109700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 0x7fc8fc15a9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4015040 tx=0x7fc8f40014a0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 shutdown_connections 2023-11-08T08:21:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8fc1577e0 0x7fc8fc15a9c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8e00682f0 0x7fc8e006a7a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8fc14b040 0x7fc8fc0a84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 --2- 172.21.15.116:0/3750636743 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8fc14b410 0x7fc8fc1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 >> 172.21.15.116:0/3750636743 conn(0x7fc8fc06ea80 msgr2=0x7fc8fc0d1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 shutdown_connections 2023-11-08T08:21:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:44.766+0000 7fc903109700 1 -- 172.21.15.116:0/3750636743 wait complete. 2023-11-08T08:21:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-08T08:21:44.777 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-08T08:21:44.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:44.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-08T08:21:44.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[146160]: fuse finished with error 0 and tester_r 0 2023-11-08T08:21:44.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:21:44.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:44.817 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-08T08:21:44.875 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:44.875 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:44.896 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:44.896 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:21:44.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:44.897 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-08T08:21:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:21:44.956 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-11-08T08:21:44.956 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:44.957 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:44.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:21:44.975 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:21:44.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:44.976 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-08T08:21:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:21:45.033 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-11-08T08:21:45.033 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:45.033 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:45.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:45.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:21:45.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:45.055 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-08T08:21:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:21:45.113 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-11-08T08:21:45.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:45.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 Processor -- start 2023-11-08T08:21:45.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 -- start start 2023-11-08T08:21:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa40406e400 0x7fa40406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa40415e010 0x7fa4040ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40415af30 con 0x7fa4040d15f0 2023-11-08T08:21:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40415a9d0 con 0x7fa40406e400 2023-11-08T08:21:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.234+0000 7fa40a71e700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4041584e0 con 0x7fa40415e010 2023-11-08T08:21:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa40406e400 0x7fa40406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa40406e400 0x7fa40406e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45998/0 (socket says 172.21.15.116:45998) 2023-11-08T08:21:45.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 -- 172.21.15.116:0/3769930969 learned_addr learned my addr 172.21.15.116:0/3769930969 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:45.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 -- 172.21.15.116:0/3769930969 >> v2:172.21.15.129:3300/0 conn(0x7fa40415e010 msgr2=0x7fa4040ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:45.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 --2- 172.21.15.116:0/3769930969 >> v2:172.21.15.129:3300/0 conn(0x7fa40415e010 0x7fa4040ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 -- 172.21.15.116:0/3769930969 >> v1:172.21.15.129:6789/0 conn(0x7fa4040d15f0 legacy=0x7fa40415b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:45.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 -- 172.21.15.116:0/3769930969 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40415ef40 con 0x7fa40406e400 2023-11-08T08:21:45.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa403fff700 1 --2- 172.21.15.116:0/3769930969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40406e400 0x7fa40406e7d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa4040b60f0 tx=0x7fa3f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d80a775fd143bf server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa402ffd700 1 -- 172.21.15.116:0/3769930969 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4004230 con 0x7fa40406e400 2023-11-08T08:21:45.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa402ffd700 1 -- 172.21.15.116:0/3769930969 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4004390 con 0x7fa40406e400 2023-11-08T08:21:45.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.235+0000 7fa402ffd700 1 -- 172.21.15.116:0/3769930969 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40036a0 con 0x7fa40406e400 2023-11-08T08:21:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 -- 172.21.15.116:0/3769930969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40406e400 msgr2=0x7fa40406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 --2- 172.21.15.116:0/3769930969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40406e400 0x7fa40406e7d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa4040b60f0 tx=0x7fa3f4009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 -- 172.21.15.116:0/3769930969 shutdown_connections 2023-11-08T08:21:45.241 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 --2- 172.21.15.116:0/3769930969 >> v2:172.21.15.129:3300/0 conn(0x7fa40415e010 0x7fa4040ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 --2- 172.21.15.116:0/3769930969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40406e400 0x7fa40406e7d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 -- 172.21.15.116:0/3769930969 >> 172.21.15.116:0/3769930969 conn(0x7fa40406a270 msgr2=0x7fa40406a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 -- 172.21.15.116:0/3769930969 shutdown_connections 2023-11-08T08:21:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.236+0000 7fa40a71e700 1 -- 172.21.15.116:0/3769930969 wait complete. 2023-11-08T08:21:45.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.237+0000 7fa40a71e700 1 Processor -- start 2023-11-08T08:21:45.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.237+0000 7fa40a71e700 1 -- start start 2023-11-08T08:21:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa40406e400 0x7fa4040cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 0x7fa404150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40415e010 0x7fa40414bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40406be30 con 0x7fa4040d15f0 2023-11-08T08:21:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40406c110 con 0x7fa40415e010 2023-11-08T08:21:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40406c3f0 con 0x7fa40406e400 2023-11-08T08:21:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 0x7fa404150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 0x7fa404150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46000/0 (socket says 172.21.15.116:46000) 2023-11-08T08:21:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 -- 172.21.15.116:0/138376038 learned_addr learned my addr 172.21.15.116:0/138376038 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa408cbb700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40415e010 0x7fa40414bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa403fff700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa40406e400 0x7fa4040cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 -- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa40406e400 msgr2=0x7fa4040cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa40406e400 0x7fa4040cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 -- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40415e010 msgr2=0x7fa40414bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40415e010 0x7fa40414bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 -- 172.21.15.116:0/138376038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4009550 con 0x7fa4040d15f0 2023-11-08T08:21:45.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa408cbb700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40415e010 0x7fa40414bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:45.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa403fff700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa40406e400 0x7fa4040cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:45.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa4037fe700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 0x7fa404150d70 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c4a0 tx=0x7fa3f800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80054e0 con 0x7fa4040d15f0 2023-11-08T08:21:45.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f8003680 con 0x7fa4040d15f0 2023-11-08T08:21:45.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40415e980 con 0x7fa4040d15f0 2023-11-08T08:21:45.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8003a10 con 0x7fa4040d15f0 2023-11-08T08:21:45.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.238+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40406ed10 con 0x7fa4040d15f0 2023-11-08T08:21:45.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.239+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40415ec60 con 0x7fa4040d15f0 2023-11-08T08:21:45.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.240+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa3f8019030 con 0x7fa4040d15f0 2023-11-08T08:21:45.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.240+0000 7fa400ff9700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa3f00682f0 0x7fa3f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.240+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa3f8084f00 con 0x7fa4040d15f0 2023-11-08T08:21:45.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.240+0000 7fa403fff700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa3f00682f0 0x7fa3f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.240+0000 7fa403fff700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa3f00682f0 0x7fa3f006a7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009c10 tx=0x7fa3f4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.244+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa3f804b5e0 con 0x7fa4040d15f0 2023-11-08T08:21:45.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.465+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa40406cbf0 con 0x7fa4040d15f0 2023-11-08T08:21:45.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.465+0000 7fa400ff9700 1 -- 172.21.15.116:0/138376038 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4661 (secure 0 0 0) 0x7fa40406cbf0 con 0x7fa4040d15f0 2023-11-08T08:21:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:45.466 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:44.504673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa3f00682f0 msgr2=0x7fa3f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa3f00682f0 0x7fa3f006a7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009c10 tx=0x7fa3f4008030 comp rx=0 tx=0).stop 2023-11-08T08:21:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 msgr2=0x7fa404150d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 0x7fa404150d70 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c4a0 tx=0x7fa3f800c860 comp rx=0 tx=0).stop 2023-11-08T08:21:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 shutdown_connections 2023-11-08T08:21:45.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fa3f00682f0 0x7fa3f006a7a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40415e010 0x7fa40414bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa40406e400 0x7fa4040cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 --2- 172.21.15.116:0/138376038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040d15f0 0x7fa404150d70 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 >> 172.21.15.116:0/138376038 conn(0x7fa40406a270 msgr2=0x7fa404153660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:45.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 shutdown_connections 2023-11-08T08:21:45.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.467+0000 7fa40a71e700 1 -- 172.21.15.116:0/138376038 wait complete. 2023-11-08T08:21:45.471 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2023-11-08T08:21:45.479 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:45.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.599+0000 7f24d0a62700 1 Processor -- start 2023-11-08T08:21:45.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.599+0000 7f24d0a62700 1 -- start start 2023-11-08T08:21:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24d0a62700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24d0a62700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24cc15cc80 0x7f24cc06de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24d0a62700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc06ab80 con 0x7f24cc06e3a0 2023-11-08T08:21:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24d0a62700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24cc0b2490 con 0x7f24cc15cc80 2023-11-08T08:21:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24d0a62700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0b1810 con 0x7f24cc06a270 2023-11-08T08:21:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24c9d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24cc15cc80 0x7f24cc06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24c9d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24cc15cc80 0x7f24cc06de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46004/0 (socket says 172.21.15.116:46004) 2023-11-08T08:21:45.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.600+0000 7f24c9d9b700 1 -- 172.21.15.116:0/1786590647 learned_addr learned my addr 172.21.15.116:0/1786590647 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:45.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24ca59c700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c9d9b700 1 -- 172.21.15.116:0/1786590647 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 msgr2=0x7f24cc06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c9d9b700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc06a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c9d9b700 1 -- 172.21.15.116:0/1786590647 >> v1:172.21.15.129:6789/0 conn(0x7f24cc06e3a0 legacy=0x7f24cc06ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c9d9b700 1 -- 172.21.15.116:0/1786590647 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc0b1af0 con 0x7f24cc15cc80 2023-11-08T08:21:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24ca59c700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc06a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c9d9b700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc06de60 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f24b40098f0 tx=0x7f24b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15caf4b627e4369d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c959a700 1 -- 172.21.15.116:0/1786590647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b400b8a0 con 0x7f24cc15cc80 2023-11-08T08:21:45.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c959a700 1 -- 172.21.15.116:0/1786590647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b400ba00 con 0x7f24cc15cc80 2023-11-08T08:21:45.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24c959a700 1 -- 172.21.15.116:0/1786590647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b400bd90 con 0x7f24cc15cc80 2023-11-08T08:21:45.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24d0a62700 1 -- 172.21.15.116:0/1786590647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 msgr2=0x7f24cc06de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.601+0000 7f24d0a62700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc06de60 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f24b40098f0 tx=0x7f24b4009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.602+0000 7f24d0a62700 1 -- 172.21.15.116:0/1786590647 shutdown_connections 2023-11-08T08:21:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.602+0000 7f24d0a62700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc06a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.602+0000 7f24d0a62700 1 --2- 172.21.15.116:0/1786590647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc06de60 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.602+0000 7f24d0a62700 1 -- 172.21.15.116:0/1786590647 >> 172.21.15.116:0/1786590647 conn(0x7f24cc06c010 msgr2=0x7f24cc0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.602+0000 7f24d0a62700 1 -- 172.21.15.116:0/1786590647 shutdown_connections 2023-11-08T08:21:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.602+0000 7f24d0a62700 1 -- 172.21.15.116:0/1786590647 wait complete. 2023-11-08T08:21:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.603+0000 7f24d0a62700 1 Processor -- start 2023-11-08T08:21:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.603+0000 7f24d0a62700 1 -- start start 2023-11-08T08:21:45.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24cc06e3a0 0x7f24cc0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc0c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0c6e60 con 0x7f24cc15cc80 2023-11-08T08:21:45.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0addd0 con 0x7f24cc06e3a0 2023-11-08T08:21:45.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0c7100 con 0x7f24cc06a270 2023-11-08T08:21:45.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc0c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc0c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46006/0 (socket says 172.21.15.116:46006) 2023-11-08T08:21:45.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 -- 172.21.15.116:0/408562872 learned_addr learned my addr 172.21.15.116:0/408562872 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:45.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24ca59c700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24c9d9b700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24cc06e3a0 0x7f24cc0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 -- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 msgr2=0x7f24cc153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 -- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24cc06e3a0 msgr2=0x7f24cc0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24cc06e3a0 0x7f24cc0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 -- 172.21.15.116:0/408562872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4009550 con 0x7f24cc15cc80 2023-11-08T08:21:45.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24cad9d700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc0c7900 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00cb50 tx=0x7f24bc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc003c00 con 0x7f24cc15cc80 2023-11-08T08:21:45.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc003d60 con 0x7f24cc15cc80 2023-11-08T08:21:45.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc00aa20 con 0x7f24cc15cc80 2023-11-08T08:21:45.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc0b20b0 con 0x7f24cc15cc80 2023-11-08T08:21:45.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.604+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc0b1dd0 con 0x7f24cc15cc80 2023-11-08T08:21:45.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.605+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc06e7e0 con 0x7f24cc15cc80 2023-11-08T08:21:45.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.606+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24bc010050 con 0x7f24cc15cc80 2023-11-08T08:21:45.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.606+0000 7f24c2ffd700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24ac0682f0 0x7f24ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.606+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f24bc085b20 con 0x7f24cc15cc80 2023-11-08T08:21:45.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.606+0000 7f24ca59c700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24ac0682f0 0x7f24ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.607+0000 7f24ca59c700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24ac0682f0 0x7f24ac06a7a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0cf460 tx=0x7f24c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.610+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24bc052380 con 0x7f24cc15cc80 2023-11-08T08:21:45.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.791+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24cc0c7620 con 0x7f24cc15cc80 2023-11-08T08:21:45.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.792+0000 7f24c2ffd700 1 -- 172.21.15.116:0/408562872 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17959 (secure 0 0 0) 0x7f24cc0c7620 con 0x7f24cc15cc80 2023-11-08T08:21:45.792 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:45.793 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:33.084884+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24ac0682f0 msgr2=0x7f24ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24ac0682f0 0x7f24ac06a7a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0cf460 tx=0x7f24c4008040 comp rx=0 tx=0).stop 2023-11-08T08:21:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 msgr2=0x7f24cc0c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc0c7900 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00cb50 tx=0x7f24bc00ce60 comp rx=0 tx=0).stop 2023-11-08T08:21:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 shutdown_connections 2023-11-08T08:21:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc15cc80 0x7f24cc0c7900 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24cc06a270 0x7f24cc153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24ac0682f0 0x7f24ac06a7a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 --2- 172.21.15.116:0/408562872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24cc06e3a0 0x7f24cc0ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 >> 172.21.15.116:0/408562872 conn(0x7f24cc06c010 msgr2=0x7f24cc0b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 shutdown_connections 2023-11-08T08:21:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.794+0000 7f24d0a62700 1 -- 172.21.15.116:0/408562872 wait complete. 2023-11-08T08:21:45.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-11-08T08:18:35.994710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-11-08T08:18:36.392792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-08T08:21:29.779880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-11-08T08:21:31.046244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-08T08:21:45.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-08T08:21:45.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.930+0000 7f7018936700 1 Processor -- start 2023-11-08T08:21:45.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.930+0000 7f7018936700 1 -- start start 2023-11-08T08:21:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.931+0000 7f7018936700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f701406c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.931+0000 7f7018936700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70140cd480 con 0x7f701415a590 2023-11-08T08:21:45.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.931+0000 7f7018936700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70140cd760 con 0x7f701415a960 2023-11-08T08:21:45.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.931+0000 7f7018936700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70140cda40 con 0x7f70140b50d0 2023-11-08T08:21:45.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.931+0000 7f701259c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f701415a590 0x7f701406b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39488/0 (socket says 172.21.15.116:39488) 2023-11-08T08:21:45.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.931+0000 7f701259c700 1 -- 172.21.15.116:0/3025492597 learned_addr learned my addr 172.21.15.116:0/3025492597 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:45.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f7011d9b700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f701406c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082213933 0 0) 0x7f70140cd480 con 0x7f701415a590 2023-11-08T08:21:45.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc0035b0 con 0x7f701415a590 2023-11-08T08:21:45.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649780863 0 0) 0x7f6ffc0035b0 con 0x7f701415a590 2023-11-08T08:21:45.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 msgr2=0x7f701406c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:45.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f701406c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 >> v1:172.21.15.129:6789/0 conn(0x7f701415a960 legacy=0x7f701406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140cdd20 con 0x7f701415a590 2023-11-08T08:21:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f7008001670 con 0x7f701415a590 2023-11-08T08:21:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 >> v1:172.21.15.116:6789/0 conn(0x7f701415a590 legacy=0x7f701406b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f6ffc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f6ffc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ffc004d90 0x7f6ffc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70140cda40 con 0x7f6ffc003a30 2023-11-08T08:21:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc00bac0 con 0x7f6ffc004d90 2023-11-08T08:21:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc00bc80 con 0x7f70140b50d0 2023-11-08T08:21:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701259c700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f6ffc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f7012d9d700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ffc004d90 0x7f6ffc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f7011d9b700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f6ffc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701259c700 1 -- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 msgr2=0x7f6ffc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701259c700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f6ffc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701259c700 1 -- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ffc004d90 msgr2=0x7f6ffc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701259c700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ffc004d90 0x7f6ffc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f701259c700 1 -- 172.21.15.116:0/3025492597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0035b0 con 0x7f6ffc003a30 2023-11-08T08:21:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.932+0000 7f7011d9b700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70140b50d0 0x7f6ffc004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f701259c700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f6ffc003e00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f700800c2f0 tx=0x7f6ffc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428a143fe44af93 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700800ce30 con 0x7f6ffc003a30 2023-11-08T08:21:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008005960 con 0x7f6ffc003a30 2023-11-08T08:21:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f701159a700 1 -- 172.21.15.116:0/3025492597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008005cf0 con 0x7f6ffc003a30 2023-11-08T08:21:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7018936700 1 -- 172.21.15.116:0/3025492597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 msgr2=0x7f6ffc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7018936700 1 --2- 172.21.15.116:0/3025492597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f6ffc003e00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f700800c2f0 tx=0x7f6ffc005880 comp rx=0 tx=0).stop 2023-11-08T08:21:45.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7012d9d700 1 -- 172.21.15.116:0/3025492597 reap_dead start 2023-11-08T08:21:45.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7018936700 1 -- 172.21.15.116:0/3025492597 shutdown_connections 2023-11-08T08:21:45.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7018936700 1 -- 172.21.15.116:0/3025492597 >> 172.21.15.116:0/3025492597 conn(0x7f70140cf400 msgr2=0x7f70140cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:45.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7018936700 1 -- 172.21.15.116:0/3025492597 shutdown_connections 2023-11-08T08:21:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.933+0000 7f7018936700 1 -- 172.21.15.116:0/3025492597 wait complete. 2023-11-08T08:21:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.935+0000 7f7018936700 1 Processor -- start 2023-11-08T08:21:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.935+0000 7f7018936700 1 -- start start 2023-11-08T08:21:45.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f7018936700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f7014151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f7018936700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70141f4890 0x7f7014152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f7018936700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f701414f090 0x7f70141f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f7018936700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7014155c40 con 0x7f6ffc003a30 2023-11-08T08:21:45.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f7018936700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7014155f20 con 0x7f701414f090 2023-11-08T08:21:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f7018936700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7014150b60 con 0x7f70141f4890 2023-11-08T08:21:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f7014151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f7014151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46028/0 (socket says 172.21.15.116:46028) 2023-11-08T08:21:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 -- 172.21.15.116:0/2717931679 learned_addr learned my addr 172.21.15.116:0/2717931679 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 -- 172.21.15.116:0/2717931679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70141f4890 msgr2=0x7f7014152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70141f4890 0x7f7014152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 -- 172.21.15.116:0/2717931679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f701414f090 msgr2=0x7f70141f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f701414f090 0x7f70141f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 -- 172.21.15.116:0/2717931679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140724c0 con 0x7f6ffc003a30 2023-11-08T08:21:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.936+0000 7f701259c700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f7014151640 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0047d0 tx=0x7f7008001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.937+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008005ce0 con 0x7f6ffc003a30 2023-11-08T08:21:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.937+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008005e40 con 0x7f6ffc003a30 2023-11-08T08:21:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.937+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70140ce000 con 0x7f6ffc003a30 2023-11-08T08:21:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.937+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008004ab0 con 0x7f6ffc003a30 2023-11-08T08:21:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.937+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7014072650 con 0x7f6ffc003a30 2023-11-08T08:21:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.938+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7014151490 con 0x7f6ffc003a30 2023-11-08T08:21:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.938+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7008024070 con 0x7f6ffc003a30 2023-11-08T08:21:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.938+0000 7f7002ffd700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ff40682f0 0x7f6ff406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.938+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7008086e30 con 0x7f6ffc003a30 2023-11-08T08:21:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.938+0000 7f7011d9b700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ff40682f0 0x7f6ff406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.939+0000 7f7011d9b700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ff40682f0 0x7f6ff406a7a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f70140c5d20 tx=0x7f7004008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:45.943+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70080536f0 con 0x7f6ffc003a30 2023-11-08T08:21:46.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.122+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7014155c40 con 0x7f6ffc003a30 2023-11-08T08:21:46.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.123+0000 7f7002ffd700 1 -- 172.21.15.116:0/2717931679 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7008007da0 con 0x7f6ffc003a30 2023-11-08T08:21:46.123 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2023-11-08T08:21:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ff40682f0 msgr2=0x7f6ff406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ff40682f0 0x7f6ff406a7a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f70140c5d20 tx=0x7f7004008040 comp rx=0 tx=0).stop 2023-11-08T08:21:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 msgr2=0x7f7014151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f7014151640 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0047d0 tx=0x7f7008001a00 comp rx=0 tx=0).stop 2023-11-08T08:21:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 shutdown_connections 2023-11-08T08:21:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f6ff40682f0 0x7f6ff406a7a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f701414f090 0x7f70141f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc003a30 0x7f7014151640 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 --2- 172.21.15.116:0/2717931679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f70141f4890 0x7f7014152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 >> 172.21.15.116:0/2717931679 conn(0x7f70140cf400 msgr2=0x7f70140ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:46.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 shutdown_connections 2023-11-08T08:21:46.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.125+0000 7f7018936700 1 -- 172.21.15.116:0/2717931679 wait complete. 2023-11-08T08:21:46.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-08T08:21:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.259+0000 7fb421dfb700 1 Processor -- start 2023-11-08T08:21:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.259+0000 7fb421dfb700 1 -- start start 2023-11-08T08:21:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb421dfb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb41c0cf710 0x7fb41c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb421dfb700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c15cc10 con 0x7fb41c071890 2023-11-08T08:21:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb421dfb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c06c9b0 con 0x7fb41c06a980 2023-11-08T08:21:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb421dfb700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41c06cc90 con 0x7fb41c0cf710 2023-11-08T08:21:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41b7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb41c0cf710 0x7fb41c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41affd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb41c071890 0x7fb41c06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39504/0 (socket says 172.21.15.116:39504) 2023-11-08T08:21:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41affd700 1 -- 172.21.15.116:0/1940427497 learned_addr learned my addr 172.21.15.116:0/1940427497 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41b7fe700 1 -- 172.21.15.116:0/1940427497 >> v1:172.21.15.129:6789/0 conn(0x7fb41c06a980 legacy=0x7fb41c160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41b7fe700 1 -- 172.21.15.116:0/1940427497 >> v1:172.21.15.116:6789/0 conn(0x7fb41c071890 legacy=0x7fb41c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41a7fc700 1 -- 172.21.15.116:0/1940427497 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523394728 0 0) 0x7fb41c15cc10 con 0x7fb41c071890 2023-11-08T08:21:46.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41b7fe700 1 -- 172.21.15.116:0/1940427497 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c06cf70 con 0x7fb41c0cf710 2023-11-08T08:21:46.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.260+0000 7fb41b7fe700 1 --2- 172.21.15.116:0/1940427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c0cf710 0x7fb41c0ce740 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb410009870 tx=0x7fb410009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64741e2174fd8978 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb41a7fc700 1 -- 172.21.15.116:0/1940427497 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410011830 con 0x7fb41c0cf710 2023-11-08T08:21:46.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb41a7fc700 1 -- 172.21.15.116:0/1940427497 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb410011990 con 0x7fb41c0cf710 2023-11-08T08:21:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb41a7fc700 1 -- 172.21.15.116:0/1940427497 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410011d20 con 0x7fb41c0cf710 2023-11-08T08:21:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 -- 172.21.15.116:0/1940427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c0cf710 msgr2=0x7fb41c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 --2- 172.21.15.116:0/1940427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c0cf710 0x7fb41c0ce740 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb410009870 tx=0x7fb410009b80 comp rx=0 tx=0).stop 2023-11-08T08:21:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 -- 172.21.15.116:0/1940427497 shutdown_connections 2023-11-08T08:21:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 --2- 172.21.15.116:0/1940427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c0cf710 0x7fb41c0ce740 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 -- 172.21.15.116:0/1940427497 >> 172.21.15.116:0/1940427497 conn(0x7fb41c06c010 msgr2=0x7fb41c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 -- 172.21.15.116:0/1940427497 shutdown_connections 2023-11-08T08:21:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.261+0000 7fb421dfb700 1 -- 172.21.15.116:0/1940427497 wait complete. 2023-11-08T08:21:46.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.262+0000 7fb421dfb700 1 Processor -- start 2023-11-08T08:21:46.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 -- start start 2023-11-08T08:21:46.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb41c06a980 0x7fb41c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 0x7fb41c0ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb41c0cf710 0x7fb41c0ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c142e70 con 0x7fb41c071890 2023-11-08T08:21:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c15f690 con 0x7fb41c06a980 2023-11-08T08:21:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb421dfb700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c15f970 con 0x7fb41c0cf710 2023-11-08T08:21:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb41affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 0x7fb41c0ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb41affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 0x7fb41c0ac8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46046/0 (socket says 172.21.15.116:46046) 2023-11-08T08:21:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.263+0000 7fb41affd700 1 -- 172.21.15.116:0/3720960032 learned_addr learned my addr 172.21.15.116:0/3720960032 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41b7fe700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb41c06a980 0x7fb41c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41bfff700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb41c0cf710 0x7fb41c0ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41affd700 1 -- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb41c0cf710 msgr2=0x7fb41c0ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41affd700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb41c0cf710 0x7fb41c0ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41affd700 1 -- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb41c06a980 msgr2=0x7fb41c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41affd700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb41c06a980 0x7fb41c0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41affd700 1 -- 172.21.15.116:0/3720960032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4100094a0 con 0x7fb41c071890 2023-11-08T08:21:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41b7fe700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb41c06a980 0x7fb41c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb41affd700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 0x7fb41c0ac8f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c520 tx=0x7fb40c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c004c80 con 0x7fb41c071890 2023-11-08T08:21:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c004e00 con 0x7fb41c071890 2023-11-08T08:21:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c06d530 con 0x7fb41c071890 2023-11-08T08:21:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c007310 con 0x7fb41c071890 2023-11-08T08:21:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.264+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c06d250 con 0x7fb41c071890 2023-11-08T08:21:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.265+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c142c20 con 0x7fb41c071890 2023-11-08T08:21:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.265+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb40c01a020 con 0x7fb41c071890 2023-11-08T08:21:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.266+0000 7fb403fff700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4080682f0 0x7fb40806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.266+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb40c085890 con 0x7fb41c071890 2023-11-08T08:21:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.266+0000 7fb41b7fe700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4080682f0 0x7fb40806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.266+0000 7fb41b7fe700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4080682f0 0x7fb40806a7a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb410000c00 tx=0x7fb41001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.270+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb40c0520f0 con 0x7fb41c071890 2023-11-08T08:21:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.449+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb41c15fe60 con 0x7fb41c071890 2023-11-08T08:21:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.450+0000 7fb403fff700 1 -- 172.21.15.116:0/3720960032 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb40c055710 con 0x7fb41c071890 2023-11-08T08:21:46.450 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-08T08:21:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4080682f0 msgr2=0x7fb40806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4080682f0 0x7fb40806a7a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb410000c00 tx=0x7fb41001c040 comp rx=0 tx=0).stop 2023-11-08T08:21:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 msgr2=0x7fb41c0ac8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 0x7fb41c0ac8f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c520 tx=0x7fb40c00c8e0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 shutdown_connections 2023-11-08T08:21:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb41c0cf710 0x7fb41c0ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4080682f0 0x7fb40806a7a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb41c06a980 0x7fb41c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 --2- 172.21.15.116:0/3720960032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb41c071890 0x7fb41c0ac8f0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 >> 172.21.15.116:0/3720960032 conn(0x7fb41c06c010 msgr2=0x7fb41c15c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 shutdown_connections 2023-11-08T08:21:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.452+0000 7fb421dfb700 1 -- 172.21.15.116:0/3720960032 wait complete. 2023-11-08T08:21:46.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-08T08:21:46.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.587+0000 7f1143ce2700 1 Processor -- start 2023-11-08T08:21:46.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.587+0000 7f1143ce2700 1 -- start start 2023-11-08T08:21:46.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f1143ce2700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f113c06cb50 0x7f113c0714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f1143ce2700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0cd480 con 0x7f113c0b50d0 2023-11-08T08:21:46.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f1143ce2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0cd760 con 0x7f113c15a590 2023-11-08T08:21:46.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f1143ce2700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c0cda40 con 0x7f113c06cb50 2023-11-08T08:21:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f114227f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f113c06cb50 0x7f113c0714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f114127d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f113c0b50d0 0x7f113c06c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39514/0 (socket says 172.21.15.116:39514) 2023-11-08T08:21:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.588+0000 7f114127d700 1 -- 172.21.15.116:0/2373003802 learned_addr learned my addr 172.21.15.116:0/2373003802 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f114227f700 1 -- 172.21.15.116:0/2373003802 >> v1:172.21.15.129:6789/0 conn(0x7f113c15a590 legacy=0x7f113c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f114227f700 1 -- 172.21.15.116:0/2373003802 >> v1:172.21.15.116:6789/0 conn(0x7f113c0b50d0 legacy=0x7f113c06c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1140a7c700 1 -- 172.21.15.116:0/2373003802 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683948234 0 0) 0x7f113c0cd480 con 0x7f113c0b50d0 2023-11-08T08:21:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f114227f700 1 -- 172.21.15.116:0/2373003802 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0cdd20 con 0x7f113c06cb50 2023-11-08T08:21:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f114227f700 1 --2- 172.21.15.116:0/2373003802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c06cb50 0x7f113c0714b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1134009d90 tx=0x7f113400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72941e50ea80443 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1140a7c700 1 -- 172.21.15.116:0/2373003802 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113400e070 con 0x7f113c06cb50 2023-11-08T08:21:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1140a7c700 1 -- 172.21.15.116:0/2373003802 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113400bb90 con 0x7f113c06cb50 2023-11-08T08:21:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1140a7c700 1 -- 172.21.15.116:0/2373003802 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134003a20 con 0x7f113c06cb50 2023-11-08T08:21:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1143ce2700 1 -- 172.21.15.116:0/2373003802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c06cb50 msgr2=0x7f113c0714b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2373003802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c06cb50 0x7f113c0714b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1134009d90 tx=0x7f113400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1143ce2700 1 -- 172.21.15.116:0/2373003802 shutdown_connections 2023-11-08T08:21:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.589+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2373003802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c06cb50 0x7f113c0714b0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.590+0000 7f1143ce2700 1 -- 172.21.15.116:0/2373003802 >> 172.21.15.116:0/2373003802 conn(0x7f113c0cf400 msgr2=0x7f113c0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.590+0000 7f1143ce2700 1 -- 172.21.15.116:0/2373003802 shutdown_connections 2023-11-08T08:21:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.590+0000 7f1143ce2700 1 -- 172.21.15.116:0/2373003802 wait complete. 2023-11-08T08:21:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 Processor -- start 2023-11-08T08:21:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 -- start start 2023-11-08T08:21:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f113c06cb50 0x7f113c15d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 0x7f113c156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f113c15a590 0x7f113c156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c157a20 con 0x7f113c0b50d0 2023-11-08T08:21:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c156ea0 con 0x7f113c15a590 2023-11-08T08:21:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.591+0000 7f1143ce2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c157180 con 0x7f113c06cb50 2023-11-08T08:21:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 0x7f113c156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 0x7f113c156340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46076/0 (socket says 172.21.15.116:46076) 2023-11-08T08:21:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 -- 172.21.15.116:0/2771464398 learned_addr learned my addr 172.21.15.116:0/2771464398 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114227f700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f113c15a590 0x7f113c156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 -- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f113c06cb50 msgr2=0x7f113c15d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f1141a7e700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f113c06cb50 0x7f113c15d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f113c06cb50 0x7f113c15d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 -- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f113c15a590 msgr2=0x7f113c156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f113c15a590 0x7f113c156880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 -- 172.21.15.116:0/2771464398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134009a70 con 0x7f113c0b50d0 2023-11-08T08:21:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114227f700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f113c15a590 0x7f113c156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f114127d700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 0x7f113c156340 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f113800c520 tx=0x7f113800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138004c80 con 0x7f113c0b50d0 2023-11-08T08:21:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1138004e00 con 0x7f113c0b50d0 2023-11-08T08:21:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c15a960 con 0x7f113c0b50d0 2023-11-08T08:21:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138007310 con 0x7f113c0b50d0 2023-11-08T08:21:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.592+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c0ce000 con 0x7f113c0b50d0 2023-11-08T08:21:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.593+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c15aaf0 con 0x7f113c0b50d0 2023-11-08T08:21:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.593+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f113801a020 con 0x7f113c0b50d0 2023-11-08T08:21:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.594+0000 7f11327fc700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11200682f0 0x7f112006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.594+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1138085890 con 0x7f113c0b50d0 2023-11-08T08:21:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.594+0000 7f1141a7e700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11200682f0 0x7f112006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.594+0000 7f1141a7e700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11200682f0 0x7f112006a7a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f113c0c84d0 tx=0x7f112c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.598+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11380521a0 con 0x7f113c0b50d0 2023-11-08T08:21:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.778+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f113c15aef0 con 0x7f113c0b50d0 2023-11-08T08:21:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.778+0000 7f11327fc700 1 -- 172.21.15.116:0/2771464398 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f113c15aef0 con 0x7f113c0b50d0 2023-11-08T08:21:46.779 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-08T08:21:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11200682f0 msgr2=0x7f112006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11200682f0 0x7f112006a7a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f113c0c84d0 tx=0x7f112c008040 comp rx=0 tx=0).stop 2023-11-08T08:21:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 msgr2=0x7f113c156340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 0x7f113c156340 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f113800c520 tx=0x7f113800c8e0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 shutdown_connections 2023-11-08T08:21:46.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f11200682f0 0x7f112006a7a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f113c15a590 0x7f113c156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f113c06cb50 0x7f113c15d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 --2- 172.21.15.116:0/2771464398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f113c0b50d0 0x7f113c156340 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 >> 172.21.15.116:0/2771464398 conn(0x7f113c0cf400 msgr2=0x7f113c06b490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:46.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 shutdown_connections 2023-11-08T08:21:46.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.780+0000 7f1143ce2700 1 -- 172.21.15.116:0/2771464398 wait complete. 2023-11-08T08:21:46.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-08T08:21:46.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.916+0000 7f607ea80700 1 Processor -- start 2023-11-08T08:21:46.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.916+0000 7f607ea80700 1 -- start start 2023-11-08T08:21:46.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607ea80700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f607806a270 0x7f607806a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607ea80700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607806abd0 con 0x7f60780718e0 2023-11-08T08:21:46.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607ea80700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078160a50 con 0x7f607815c810 2023-11-08T08:21:46.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607ea80700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607815cbe0 con 0x7f607806a270 2023-11-08T08:21:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607d01d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f607806a270 0x7f607806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f6077fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f60780718e0 0x7f607806c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39520/0 (socket says 172.21.15.116:39520) 2023-11-08T08:21:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f6077fff700 1 -- 172.21.15.116:0/277126066 learned_addr learned my addr 172.21.15.116:0/277126066 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607d01d700 1 -- 172.21.15.116:0/277126066 >> v1:172.21.15.129:6789/0 conn(0x7f607815c810 legacy=0x7f6078160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607d01d700 1 -- 172.21.15.116:0/277126066 >> v1:172.21.15.116:6789/0 conn(0x7f60780718e0 legacy=0x7f607806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607d01d700 1 -- 172.21.15.116:0/277126066 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078160d30 con 0x7f607806a270 2023-11-08T08:21:46.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f60777fe700 1 -- 172.21.15.116:0/277126066 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179952764 0 0) 0x7f607806abd0 con 0x7f60780718e0 2023-11-08T08:21:46.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f607d01d700 1 --2- 172.21.15.116:0/277126066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f607806a690 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6070009dc0 tx=0x7f607000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf477ac83ea99e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f60777fe700 1 -- 172.21.15.116:0/277126066 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607000e070 con 0x7f607806a270 2023-11-08T08:21:46.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f60777fe700 1 -- 172.21.15.116:0/277126066 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607000bb50 con 0x7f607806a270 2023-11-08T08:21:46.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.917+0000 7f60777fe700 1 -- 172.21.15.116:0/277126066 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070003a20 con 0x7f607806a270 2023-11-08T08:21:46.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 -- 172.21.15.116:0/277126066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 msgr2=0x7f607806a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 --2- 172.21.15.116:0/277126066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f607806a690 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6070009dc0 tx=0x7f607000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:46.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 -- 172.21.15.116:0/277126066 shutdown_connections 2023-11-08T08:21:46.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 --2- 172.21.15.116:0/277126066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f607806a690 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 -- 172.21.15.116:0/277126066 >> 172.21.15.116:0/277126066 conn(0x7f607806c010 msgr2=0x7f60780cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:46.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 -- 172.21.15.116:0/277126066 shutdown_connections 2023-11-08T08:21:46.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.918+0000 7f607ea80700 1 -- 172.21.15.116:0/277126066 wait complete. 2023-11-08T08:21:46.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.919+0000 7f607ea80700 1 Processor -- start 2023-11-08T08:21:46.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.919+0000 7f607ea80700 1 -- start start 2023-11-08T08:21:46.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607ea80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f60780b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607ea80700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60780718e0 0x7f6078155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607ea80700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f607815c810 0x7f60780aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607ea80700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780b5900 con 0x7f607806a270 2023-11-08T08:21:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607ea80700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607815e110 con 0x7f60780718e0 2023-11-08T08:21:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607ea80700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607815e3f0 con 0x7f607815c810 2023-11-08T08:21:46.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f60780b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f60780b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46088/0 (socket says 172.21.15.116:46088) 2023-11-08T08:21:46.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 -- 172.21.15.116:0/3458619989 learned_addr learned my addr 172.21.15.116:0/3458619989 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:46.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 -- 172.21.15.116:0/3458619989 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f607815c810 msgr2=0x7f60780aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f607815c810 0x7f60780aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 -- 172.21.15.116:0/3458619989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60780718e0 msgr2=0x7f6078155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:46.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60780718e0 0x7f6078155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:46.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.920+0000 7f607c81c700 1 -- 172.21.15.116:0/3458619989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6070009a70 con 0x7f607806a270 2023-11-08T08:21:46.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.921+0000 7f607c81c700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f60780b53c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f606c00b4a0 tx=0x7f606c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.921+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c011830 con 0x7f607806a270 2023-11-08T08:21:46.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.921+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c011990 con 0x7f607806a270 2023-11-08T08:21:46.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.921+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60780ceb20 con 0x7f607806a270 2023-11-08T08:21:46.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.921+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c011d40 con 0x7f607806a270 2023-11-08T08:21:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.921+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6078161010 con 0x7f607806a270 2023-11-08T08:21:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.922+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607806cf90 con 0x7f607806a270 2023-11-08T08:21:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.922+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f606c00f050 con 0x7f607806a270 2023-11-08T08:21:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.923+0000 7f60757fa700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f605c0682f0 0x7f605c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.923+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f606c085e30 con 0x7f607806a270 2023-11-08T08:21:46.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.923+0000 7f6077fff700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f605c0682f0 0x7f605c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:46.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.923+0000 7f6077fff700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f605c0682f0 0x7f605c06a7a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f60640011e0 tx=0x7f60640079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:46.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:46.927+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f606c052620 con 0x7f607806a270 2023-11-08T08:21:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.105+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6078068450 con 0x7f607806a270 2023-11-08T08:21:47.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.106+0000 7f60757fa700 1 -- 172.21.15.116:0/3458619989 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f606c015020 con 0x7f607806a270 2023-11-08T08:21:47.106 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-08T08:21:47.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.107+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f605c0682f0 msgr2=0x7f605c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.107+0000 7f607ea80700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f605c0682f0 0x7f605c06a7a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f60640011e0 tx=0x7f60640079b0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.107+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 msgr2=0x7f60780b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.107+0000 7f607ea80700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f60780b53c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f606c00b4a0 tx=0x7f606c00b860 comp rx=0 tx=0).stop 2023-11-08T08:21:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 shutdown_connections 2023-11-08T08:21:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f607815c810 0x7f60780aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f607806a270 0x7f60780b53c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f605c0682f0 0x7f605c06a7a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 --2- 172.21.15.116:0/3458619989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60780718e0 0x7f6078155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 >> 172.21.15.116:0/3458619989 conn(0x7f607806c010 msgr2=0x7f60780ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:47.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 shutdown_connections 2023-11-08T08:21:47.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.108+0000 7f607ea80700 1 -- 172.21.15.116:0/3458619989 wait complete. 2023-11-08T08:21:47.119 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 Processor -- start 2023-11-08T08:21:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 -- start start 2023-11-08T08:21:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f244406a270 0x7f244406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 0x7f244406de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244406ab80 con 0x7f244406e3a0 2023-11-08T08:21:47.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24440b2490 con 0x7f244406a270 2023-11-08T08:21:47.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.240+0000 7f244a3c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24440b1810 con 0x7f244415cc80 2023-11-08T08:21:47.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f244895e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f244406e3a0 0x7f244406ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39522/0 (socket says 172.21.15.116:39522) 2023-11-08T08:21:47.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f244895e700 1 -- 172.21.15.116:0/1108668718 learned_addr learned my addr 172.21.15.116:0/1108668718 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:47.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f24437fe700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 0x7f244406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2443fff700 1 --2- 172.21.15.116:0/1108668718 >> v2:172.21.15.129:3300/0 conn(0x7f244406a270 0x7f244406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906181621 0 0) 0x7f244406ab80 con 0x7f244406e3a0 2023-11-08T08:21:47.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242c0035b0 con 0x7f244406e3a0 2023-11-08T08:21:47.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928757969 0 0) 0x7f242c0035b0 con 0x7f244406e3a0 2023-11-08T08:21:47.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 msgr2=0x7f244406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 0x7f244406de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 >> v2:172.21.15.129:3300/0 conn(0x7f244406a270 msgr2=0x7f244406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 --2- 172.21.15.116:0/1108668718 >> v2:172.21.15.129:3300/0 conn(0x7f244406a270 0x7f244406a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24440b1af0 con 0x7f244406e3a0 2023-11-08T08:21:47.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f243c0019f0 con 0x7f244406e3a0 2023-11-08T08:21:47.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 >> v1:172.21.15.116:6789/0 conn(0x7f244406e3a0 legacy=0x7f244406ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f242c003a00 0x7f242c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c004430 0x7f242c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 0x7f242c005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24440b2490 con 0x7f242c004430 2023-11-08T08:21:47.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24440b1810 con 0x7f242c003a00 2023-11-08T08:21:47.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c00ba10 con 0x7f244415cc80 2023-11-08T08:21:47.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f24437fe700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c004430 0x7f242c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f244895e700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 0x7f242c005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f2443fff700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f242c003a00 0x7f242c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f24437fe700 1 -- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 msgr2=0x7f242c005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f24437fe700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f244415cc80 0x7f242c005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f24437fe700 1 -- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f242c003a00 msgr2=0x7f242c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.241+0000 7f24437fe700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f242c003a00 0x7f242c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f24437fe700 1 -- 172.21.15.116:0/1108668718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0035b0 con 0x7f242c004430 2023-11-08T08:21:47.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f2443fff700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f242c003a00 0x7f242c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:47.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f24437fe700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c004430 0x7f242c0048f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2438009b90 tx=0x7f242c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d03a15d9ea0d31f server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243800e6c0 con 0x7f242c004430 2023-11-08T08:21:47.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243800e820 con 0x7f242c004430 2023-11-08T08:21:47.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f2442ffd700 1 -- 172.21.15.116:0/1108668718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243800eb90 con 0x7f242c004430 2023-11-08T08:21:47.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244a3c1700 1 -- 172.21.15.116:0/1108668718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c004430 msgr2=0x7f242c0048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244a3c1700 1 --2- 172.21.15.116:0/1108668718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f242c004430 0x7f242c0048f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2438009b90 tx=0x7f242c005880 comp rx=0 tx=0).stop 2023-11-08T08:21:47.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244895e700 1 -- 172.21.15.116:0/1108668718 reap_dead start 2023-11-08T08:21:47.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244a3c1700 1 -- 172.21.15.116:0/1108668718 shutdown_connections 2023-11-08T08:21:47.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244a3c1700 1 -- 172.21.15.116:0/1108668718 >> 172.21.15.116:0/1108668718 conn(0x7f244406c010 msgr2=0x7f24440cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:47.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244a3c1700 1 -- 172.21.15.116:0/1108668718 shutdown_connections 2023-11-08T08:21:47.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.242+0000 7f244a3c1700 1 -- 172.21.15.116:0/1108668718 wait complete. 2023-11-08T08:21:47.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.244+0000 7f244a3c1700 1 Processor -- start 2023-11-08T08:21:47.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.244+0000 7f244a3c1700 1 -- start start 2023-11-08T08:21:47.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244a3c1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f242c004430 0x7f24440c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244a3c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 0x7f2444152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244a3c1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2444152460 0x7f24440cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244a3c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24440c7790 con 0x7f24440c7c20 2023-11-08T08:21:47.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244a3c1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24440c6e60 con 0x7f2444152460 2023-11-08T08:21:47.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244a3c1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24440c7140 con 0x7f242c004430 2023-11-08T08:21:47.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 0x7f2444152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 0x7f2444152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46098/0 (socket says 172.21.15.116:46098) 2023-11-08T08:21:47.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 -- 172.21.15.116:0/774669916 learned_addr learned my addr 172.21.15.116:0/774669916 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:47.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f2443fff700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f242c004430 0x7f24440c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f244895e700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2444152460 0x7f24440cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 -- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f242c004430 msgr2=0x7f24440c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f242c004430 0x7f24440c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 -- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2444152460 msgr2=0x7f24440cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2444152460 0x7f24440cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f24437fe700 1 -- 172.21.15.116:0/774669916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24440b20b0 con 0x7f24440c7c20 2023-11-08T08:21:47.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.245+0000 7f2443fff700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f242c004430 0x7f24440c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:47.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.246+0000 7f24437fe700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 0x7f2444152c60 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f242c005770 tx=0x7f24380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.246+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243800e6c0 con 0x7f24440c7c20 2023-11-08T08:21:47.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.246+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24380038a0 con 0x7f24440c7c20 2023-11-08T08:21:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.246+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24440b1dd0 con 0x7f24440c7c20 2023-11-08T08:21:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.246+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438003c50 con 0x7f24440c7c20 2023-11-08T08:21:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.246+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244406e7e0 con 0x7f24440c7c20 2023-11-08T08:21:47.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.247+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24440c7900 con 0x7f24440c7c20 2023-11-08T08:21:47.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.247+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2438019030 con 0x7f24440c7c20 2023-11-08T08:21:47.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.247+0000 7f2440ff9700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24300682f0 0x7f243006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.248+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2438087260 con 0x7f24440c7c20 2023-11-08T08:21:47.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.248+0000 7f2443fff700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24300682f0 0x7f243006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.248+0000 7f2443fff700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24300682f0 0x7f243006a7a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f244415bff0 tx=0x7f2434008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.252+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2438053ac0 con 0x7f24440c7c20 2023-11-08T08:21:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.474+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24440743f0 con 0x7f24440c7c20 2023-11-08T08:21:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.475+0000 7f2440ff9700 1 -- 172.21.15.116:0/774669916 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4661 (secure 0 0 0) 0x7f24380570e0 con 0x7f24440c7c20 2023-11-08T08:21:47.476 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:47.476 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:44.504673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24300682f0 msgr2=0x7f243006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24300682f0 0x7f243006a7a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f244415bff0 tx=0x7f2434008040 comp rx=0 tx=0).stop 2023-11-08T08:21:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 msgr2=0x7f2444152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 0x7f2444152c60 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f242c005770 tx=0x7f24380047c0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 shutdown_connections 2023-11-08T08:21:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f24300682f0 0x7f243006a7a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2444152460 0x7f24440cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f242c004430 0x7f24440c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 --2- 172.21.15.116:0/774669916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24440c7c20 0x7f2444152c60 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 >> 172.21.15.116:0/774669916 conn(0x7f244406c010 msgr2=0x7f24440afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 shutdown_connections 2023-11-08T08:21:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.477+0000 7f244a3c1700 1 -- 172.21.15.116:0/774669916 wait complete. 2023-11-08T08:21:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2023-11-08T08:21:47.489 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:47.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.609+0000 7fc8d8515700 1 Processor -- start 2023-11-08T08:21:47.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.609+0000 7fc8d8515700 1 -- start start 2023-11-08T08:21:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.610+0000 7fc8d8515700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8d0070d90 0x7fc8d00711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.610+0000 7fc8d8515700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d0151a50 con 0x7fc8d015e0b0 2023-11-08T08:21:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.610+0000 7fc8d8515700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d015b440 con 0x7fc8d006a270 2023-11-08T08:21:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.610+0000 7fc8d8515700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d0071ac0 con 0x7fc8d0070d90 2023-11-08T08:21:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.610+0000 7fc8d5ab0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8d0070d90 0x7fc8d00711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.610+0000 7fc8d6ab2700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8d015e0b0 0x7fc8d015e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39524/0 (socket says 172.21.15.116:39524) 2023-11-08T08:21:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d6ab2700 1 -- 172.21.15.116:0/251228693 learned_addr learned my addr 172.21.15.116:0/251228693 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d5ab0700 1 -- 172.21.15.116:0/251228693 >> v1:172.21.15.129:6789/0 conn(0x7fc8d006a270 legacy=0x7fc8d006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d5ab0700 1 -- 172.21.15.116:0/251228693 >> v1:172.21.15.116:6789/0 conn(0x7fc8d015e0b0 legacy=0x7fc8d015e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d5ab0700 1 -- 172.21.15.116:0/251228693 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d015adf0 con 0x7fc8d0070d90 2023-11-08T08:21:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d52af700 1 -- 172.21.15.116:0/251228693 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099411831 0 0) 0x7fc8d0151a50 con 0x7fc8d015e0b0 2023-11-08T08:21:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d5ab0700 1 --2- 172.21.15.116:0/251228693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0070d90 0x7fc8d00711e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009840 tx=0x7fc8c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a49cf89be8bb3a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d52af700 1 -- 172.21.15.116:0/251228693 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4004230 con 0x7fc8d0070d90 2023-11-08T08:21:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d52af700 1 -- 172.21.15.116:0/251228693 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c4004390 con 0x7fc8d0070d90 2023-11-08T08:21:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.611+0000 7fc8d52af700 1 -- 172.21.15.116:0/251228693 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40036a0 con 0x7fc8d0070d90 2023-11-08T08:21:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 -- 172.21.15.116:0/251228693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0070d90 msgr2=0x7fc8d00711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 --2- 172.21.15.116:0/251228693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0070d90 0x7fc8d00711e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009840 tx=0x7fc8c4009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 -- 172.21.15.116:0/251228693 shutdown_connections 2023-11-08T08:21:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 --2- 172.21.15.116:0/251228693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0070d90 0x7fc8d00711e0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 -- 172.21.15.116:0/251228693 >> 172.21.15.116:0/251228693 conn(0x7fc8d006c010 msgr2=0x7fc8d00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 -- 172.21.15.116:0/251228693 shutdown_connections 2023-11-08T08:21:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.612+0000 7fc8d8515700 1 -- 172.21.15.116:0/251228693 wait complete. 2023-11-08T08:21:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.613+0000 7fc8d8515700 1 Processor -- start 2023-11-08T08:21:47.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.613+0000 7fc8d8515700 1 -- start start 2023-11-08T08:21:47.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8d006a270 0x7fc8d01f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8d0070d90 0x7fc8d0146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 0x7fc8d00cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00c3940 con 0x7fc8d015e0b0 2023-11-08T08:21:47.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00c3c20 con 0x7fc8d006a270 2023-11-08T08:21:47.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00c3f00 con 0x7fc8d0070d90 2023-11-08T08:21:47.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 0x7fc8d00cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 0x7fc8d00cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46116/0 (socket says 172.21.15.116:46116) 2023-11-08T08:21:47.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 -- 172.21.15.116:0/1542868629 learned_addr learned my addr 172.21.15.116:0/1542868629 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:47.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d62b1700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8d006a270 0x7fc8d01f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d5ab0700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8d0070d90 0x7fc8d0146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 -- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8d0070d90 msgr2=0x7fc8d0146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8d0070d90 0x7fc8d0146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 -- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8d006a270 msgr2=0x7fc8d01f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8d006a270 0x7fc8d01f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 -- 172.21.15.116:0/1542868629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40094a0 con 0x7fc8d015e0b0 2023-11-08T08:21:47.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d5ab0700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8d0070d90 0x7fc8d0146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:47.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d6ab2700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 0x7fc8d00cecc0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800cb60 tx=0x7fc8c800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800ab50 con 0x7fc8d015e0b0 2023-11-08T08:21:47.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c800acb0 con 0x7fc8d015e0b0 2023-11-08T08:21:47.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d015aac0 con 0x7fc8d015e0b0 2023-11-08T08:21:47.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8007310 con 0x7fc8d015e0b0 2023-11-08T08:21:47.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.614+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d015b0d0 con 0x7fc8d015e0b0 2023-11-08T08:21:47.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.615+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d0071720 con 0x7fc8d015e0b0 2023-11-08T08:21:47.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.615+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8c801a030 con 0x7fc8d015e0b0 2023-11-08T08:21:47.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.616+0000 7fc8ceffd700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8bc0682f0 0x7fc8bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.616+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc8c8085940 con 0x7fc8d015e0b0 2023-11-08T08:21:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.616+0000 7fc8d62b1700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8bc0682f0 0x7fc8bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.616+0000 7fc8d62b1700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8bc0682f0 0x7fc8bc06a7a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc8d015d170 tx=0x7fc8c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.620+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8c80521a0 con 0x7fc8d015e0b0 2023-11-08T08:21:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.804+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8d00c3d60 con 0x7fc8d015e0b0 2023-11-08T08:21:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.805+0000 7fc8ceffd700 1 -- 172.21.15.116:0/1542868629 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17959 (secure 0 0 0) 0x7fc8d00c3d60 con 0x7fc8d015e0b0 2023-11-08T08:21:47.806 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:47.806 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:33.084884+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8bc0682f0 msgr2=0x7fc8bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8bc0682f0 0x7fc8bc06a7a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc8d015d170 tx=0x7fc8c0009040 comp rx=0 tx=0).stop 2023-11-08T08:21:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 msgr2=0x7fc8d00cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 0x7fc8d00cecc0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800cb60 tx=0x7fc8c800cf20 comp rx=0 tx=0).stop 2023-11-08T08:21:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 shutdown_connections 2023-11-08T08:21:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d015e0b0 0x7fc8d00cecc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc8bc0682f0 0x7fc8bc06a7a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8d006a270 0x7fc8d01f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 --2- 172.21.15.116:0/1542868629 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc8d0070d90 0x7fc8d0146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 >> 172.21.15.116:0/1542868629 conn(0x7fc8d006c010 msgr2=0x7fc8d00b5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 shutdown_connections 2023-11-08T08:21:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.807+0000 7fc8d8515700 1 -- 172.21.15.116:0/1542868629 wait complete. 2023-11-08T08:21:47.820 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-08T08:21:47.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.940+0000 7f1f0e245700 1 Processor -- start 2023-11-08T08:21:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.940+0000 7f1f0e245700 1 -- start start 2023-11-08T08:21:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f0e245700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 0x7f1f080b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f0e245700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0806e900 con 0x7f1f08160e70 2023-11-08T08:21:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f0e245700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f08160390 con 0x7f1f080d0c60 2023-11-08T08:21:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f0e245700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f08160670 con 0x7f1f0806e400 2023-11-08T08:21:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f07fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1f08160e70 0x7f1f0815c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39536/0 (socket says 172.21.15.116:39536) 2023-11-08T08:21:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f07fff700 1 -- 172.21.15.116:0/3617160392 learned_addr learned my addr 172.21.15.116:0/3617160392 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f06ffd700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 0x7f1f080b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818511349 0 0) 0x7f1f0806e900 con 0x7f1f08160e70 2023-11-08T08:21:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef00035b0 con 0x7f1f08160e70 2023-11-08T08:21:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854935389 0 0) 0x7f1ef00035b0 con 0x7f1f08160e70 2023-11-08T08:21:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 msgr2=0x7f1f080b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 0x7f1f080b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 >> v1:172.21.15.129:6789/0 conn(0x7f1f080d0c60 legacy=0x7f1f080d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08160950 con 0x7f1f08160e70 2023-11-08T08:21:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f1f000019f0 con 0x7f1f08160e70 2023-11-08T08:21:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 >> v1:172.21.15.116:6789/0 conn(0x7f1f08160e70 legacy=0x7f1f0815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 0x7f1ef0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ef0003f70 0x7f1ef0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0004d90 0x7f1ef0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f08160670 con 0x7f1ef0004d90 2023-11-08T08:21:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.941+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef000ba40 con 0x7f1ef0003f70 2023-11-08T08:21:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef000bc00 con 0x7f1f0806e400 2023-11-08T08:21:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0004d90 0x7f1ef0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f077fe700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 0x7f1ef0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f06ffd700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ef0003f70 0x7f1ef0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 -- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 msgr2=0x7f1ef0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f0806e400 0x7f1ef0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 -- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ef0003f70 msgr2=0x7f1ef0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ef0003f70 0x7f1ef0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 -- 172.21.15.116:0/3617160392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00035b0 con 0x7f1ef0004d90 2023-11-08T08:21:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f07fff700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0004d90 0x7f1ef0005200 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000a7a0 tx=0x7f1ef0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7146cb7bf9ec48 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00012600 con 0x7f1ef0004d90 2023-11-08T08:21:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00012760 con 0x7f1ef0004d90 2023-11-08T08:21:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.942+0000 7f1f067fc700 1 -- 172.21.15.116:0/3617160392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00012b10 con 0x7f1ef0004d90 2023-11-08T08:21:47.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f0e245700 1 -- 172.21.15.116:0/3617160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0004d90 msgr2=0x7f1ef0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f0e245700 1 --2- 172.21.15.116:0/3617160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0004d90 0x7f1ef0005200 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000a7a0 tx=0x7f1ef0005880 comp rx=0 tx=0).stop 2023-11-08T08:21:47.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f07fff700 1 -- 172.21.15.116:0/3617160392 reap_dead start 2023-11-08T08:21:47.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f0e245700 1 -- 172.21.15.116:0/3617160392 shutdown_connections 2023-11-08T08:21:47.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f0e245700 1 -- 172.21.15.116:0/3617160392 >> 172.21.15.116:0/3617160392 conn(0x7f1f0806a270 msgr2=0x7f1f0806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:47.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f0e245700 1 -- 172.21.15.116:0/3617160392 shutdown_connections 2023-11-08T08:21:47.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.943+0000 7f1f0e245700 1 -- 172.21.15.116:0/3617160392 wait complete. 2023-11-08T08:21:47.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.944+0000 7f1f0e245700 1 Processor -- start 2023-11-08T08:21:47.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.944+0000 7f1f0e245700 1 -- start start 2023-11-08T08:21:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f081517b0 0x7f1f080c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f08151b80 0x7f1f080ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 0x7f1f080cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080b3030 con 0x7f1f080b34c0 2023-11-08T08:21:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f081e8270 con 0x7f1f081517b0 2023-11-08T08:21:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080b29e0 con 0x7f1f08151b80 2023-11-08T08:21:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 0x7f1f080cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 0x7f1f080cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46136/0 (socket says 172.21.15.116:46136) 2023-11-08T08:21:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 -- 172.21.15.116:0/202277623 learned_addr learned my addr 172.21.15.116:0/202277623 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f06ffd700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f08151b80 0x7f1f080ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f077fe700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f081517b0 0x7f1f080c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 -- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f08151b80 msgr2=0x7f1f080ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f08151b80 0x7f1f080ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 -- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f081517b0 msgr2=0x7f1f080c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f081517b0 0x7f1f080c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 -- 172.21.15.116:0/202277623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080b5be0 con 0x7f1f080b34c0 2023-11-08T08:21:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f07fff700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 0x7f1f080cc350 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1f00012000 tx=0x7f1f00001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f000019f0 con 0x7f1f080b34c0 2023-11-08T08:21:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0001e430 con 0x7f1f080b34c0 2023-11-08T08:21:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f080d0740 con 0x7f1f080b34c0 2023-11-08T08:21:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.945+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0001e7e0 con 0x7f1f080b34c0 2023-11-08T08:21:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.946+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f080b5d70 con 0x7f1f080b34c0 2023-11-08T08:21:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.946+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f080b2840 con 0x7f1f080b34c0 2023-11-08T08:21:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.947+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f00010070 con 0x7f1f080b34c0 2023-11-08T08:21:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.947+0000 7f1eeffff700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ef40682f0 0x7f1ef406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.947+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1f00087910 con 0x7f1f080b34c0 2023-11-08T08:21:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.947+0000 7f1f077fe700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ef40682f0 0x7f1ef406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.948+0000 7f1f077fe700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ef40682f0 0x7f1ef406a7a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1efc007fd0 tx=0x7f1efc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:47.951+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f080b2840 con 0x7f1f080b34c0 2023-11-08T08:21:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.176+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f080b2840 con 0x7f1f080b34c0 2023-11-08T08:21:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.176+0000 7f1eeffff700 1 -- 172.21.15.116:0/202277623 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4661 (secure 0 0 0) 0x7f1f080b2840 con 0x7f1f080b34c0 2023-11-08T08:21:48.178 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:48.178 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:44.504673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.178+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ef40682f0 msgr2=0x7f1ef406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.178+0000 7f1f0e245700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ef40682f0 0x7f1ef406a7a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1efc007fd0 tx=0x7f1efc009040 comp rx=0 tx=0).stop 2023-11-08T08:21:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.178+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 msgr2=0x7f1f080cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.178+0000 7f1f0e245700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 0x7f1f080cc350 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1f00012000 tx=0x7f1f00001d00 comp rx=0 tx=0).stop 2023-11-08T08:21:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 shutdown_connections 2023-11-08T08:21:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f080b34c0 0x7f1f080cc350 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1ef40682f0 0x7f1ef406a7a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f081517b0 0x7f1f080c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 --2- 172.21.15.116:0/202277623 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1f08151b80 0x7f1f080ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 >> 172.21.15.116:0/202277623 conn(0x7f1f0806a270 msgr2=0x7f1f0806ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 shutdown_connections 2023-11-08T08:21:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.179+0000 7f1f0e245700 1 -- 172.21.15.116:0/202277623 wait complete. 2023-11-08T08:21:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2023-11-08T08:21:48.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:21:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.311+0000 7fc0a8e23700 1 Processor -- start 2023-11-08T08:21:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.312+0000 7fc0a8e23700 1 -- start start 2023-11-08T08:21:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.312+0000 7fc0a8e23700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc0a40b4660 0x7fc0a4071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.312+0000 7fc0a8e23700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc0a40cff90 0x7fc0a406b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.312+0000 7fc0a8e23700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40cfb00 con 0x7fc0a40b4a30 2023-11-08T08:21:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.312+0000 7fc0a8e23700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a415aa20 con 0x7fc0a40cff90 2023-11-08T08:21:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.312+0000 7fc0a8e23700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a40cf4d0 con 0x7fc0a40b4660 2023-11-08T08:21:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc0a40cff90 0x7fc0a406b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc0a40cff90 0x7fc0a406b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46138/0 (socket says 172.21.15.116:46138) 2023-11-08T08:21:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 -- 172.21.15.116:0/611118526 learned_addr learned my addr 172.21.15.116:0/611118526 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 -- 172.21.15.116:0/611118526 >> v2:172.21.15.129:3300/0 conn(0x7fc0a40b4660 msgr2=0x7fc0a4071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a259c700 1 --2- 172.21.15.116:0/611118526 >> v2:172.21.15.129:3300/0 conn(0x7fc0a40b4660 0x7fc0a4071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 --2- 172.21.15.116:0/611118526 >> v2:172.21.15.129:3300/0 conn(0x7fc0a40b4660 0x7fc0a4071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 -- 172.21.15.116:0/611118526 >> v1:172.21.15.129:6789/0 conn(0x7fc0a40b4a30 legacy=0x7fc0a40ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 -- 172.21.15.116:0/611118526 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a4072550 con 0x7fc0a40cff90 2023-11-08T08:21:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a259c700 1 --2- 172.21.15.116:0/611118526 >> v2:172.21.15.129:3300/0 conn(0x7fc0a40b4660 0x7fc0a4071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a1d9b700 1 --2- 172.21.15.116:0/611118526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40cff90 0x7fc0a406b590 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc0a406efd0 tx=0x7fc094009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa21b3586c88a86 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a159a700 1 -- 172.21.15.116:0/611118526 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094004230 con 0x7fc0a40cff90 2023-11-08T08:21:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a159a700 1 -- 172.21.15.116:0/611118526 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc094004390 con 0x7fc0a40cff90 2023-11-08T08:21:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.313+0000 7fc0a159a700 1 -- 172.21.15.116:0/611118526 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0940036a0 con 0x7fc0a40cff90 2023-11-08T08:21:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 -- 172.21.15.116:0/611118526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40cff90 msgr2=0x7fc0a406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/611118526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40cff90 0x7fc0a406b590 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc0a406efd0 tx=0x7fc094009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 -- 172.21.15.116:0/611118526 shutdown_connections 2023-11-08T08:21:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/611118526 >> v2:172.21.15.129:3300/0 conn(0x7fc0a40b4660 0x7fc0a4071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/611118526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40cff90 0x7fc0a406b590 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 -- 172.21.15.116:0/611118526 >> 172.21.15.116:0/611118526 conn(0x7fc0a40b2fc0 msgr2=0x7fc0a4071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 -- 172.21.15.116:0/611118526 shutdown_connections 2023-11-08T08:21:48.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.314+0000 7fc0a8e23700 1 -- 172.21.15.116:0/611118526 wait complete. 2023-11-08T08:21:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.315+0000 7fc0a8e23700 1 Processor -- start 2023-11-08T08:21:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.315+0000 7fc0a8e23700 1 -- start start 2023-11-08T08:21:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a8e23700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 0x7fc0a415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a8e23700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a40b4a30 0x7fc0a40c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a8e23700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a40cff90 0x7fc0a40c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a8e23700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a415b1e0 con 0x7fc0a40b4660 2023-11-08T08:21:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a8e23700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40c6520 con 0x7fc0a40cff90 2023-11-08T08:21:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a8e23700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40c6770 con 0x7fc0a40b4a30 2023-11-08T08:21:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 0x7fc0a415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 0x7fc0a415aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46152/0 (socket says 172.21.15.116:46152) 2023-11-08T08:21:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 -- 172.21.15.116:0/4208117010 learned_addr learned my addr 172.21.15.116:0/4208117010 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a2d9d700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a40cff90 0x7fc0a40c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a1d9b700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a40b4a30 0x7fc0a40c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 -- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a40b4a30 msgr2=0x7fc0a40c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a40b4a30 0x7fc0a40c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 -- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a40cff90 msgr2=0x7fc0a40c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a40cff90 0x7fc0a40c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 -- 172.21.15.116:0/4208117010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc094009550 con 0x7fc0a40b4660 2023-11-08T08:21:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a1d9b700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a40b4a30 0x7fc0a40c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.316+0000 7fc0a259c700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 0x7fc0a415aca0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc09800c750 tx=0x7fc09800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.317+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098011800 con 0x7fc0a40b4660 2023-11-08T08:21:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.317+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc098011960 con 0x7fc0a40b4660 2023-11-08T08:21:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.317+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a406bad0 con 0x7fc0a40b4660 2023-11-08T08:21:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.317+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098011cd0 con 0x7fc0a40b4660 2023-11-08T08:21:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.317+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a40cf790 con 0x7fc0a40b4660 2023-11-08T08:21:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.317+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a40b4e90 con 0x7fc0a40b4660 2023-11-08T08:21:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.318+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc098020070 con 0x7fc0a40b4660 2023-11-08T08:21:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.318+0000 7fc092ffd700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0840682f0 0x7fc08406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.318+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc098085a70 con 0x7fc0a40b4660 2023-11-08T08:21:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.318+0000 7fc0a1d9b700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0840682f0 0x7fc08406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.319+0000 7fc0a1d9b700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0840682f0 0x7fc08406a7a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc094009130 tx=0x7fc094008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.323+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc0980522d0 con 0x7fc0a40b4660 2023-11-08T08:21:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.547+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0a40c6b60 con 0x7fc0a40b4660 2023-11-08T08:21:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.548+0000 7fc092ffd700 1 -- 172.21.15.116:0/4208117010 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4661 (secure 0 0 0) 0x7fc0980558f0 con 0x7fc0a40b4660 2023-11-08T08:21:48.548 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:48.548 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-08T08:21:32.082988+0000","modified":"2023-11-08T08:21:44.504673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/4191005747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4191005747},{"type":"v1","addr":"172.21.15.129:6835","nonce":4191005747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-08T08:21:48.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0840682f0 msgr2=0x7fc08406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0840682f0 0x7fc08406a7a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc094009130 tx=0x7fc094008030 comp rx=0 tx=0).stop 2023-11-08T08:21:48.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 msgr2=0x7fc0a415aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 0x7fc0a415aca0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc09800c750 tx=0x7fc09800ca60 comp rx=0 tx=0).stop 2023-11-08T08:21:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 shutdown_connections 2023-11-08T08:21:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc0840682f0 0x7fc08406a7a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0a40cff90 0x7fc0a40c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a40b4660 0x7fc0a415aca0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 --2- 172.21.15.116:0/4208117010 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc0a40b4a30 0x7fc0a40c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 >> 172.21.15.116:0/4208117010 conn(0x7fc0a40b2fc0 msgr2=0x7fc0a406da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:48.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 shutdown_connections 2023-11-08T08:21:48.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.550+0000 7fc0a8e23700 1 -- 172.21.15.116:0/4208117010 wait complete. 2023-11-08T08:21:48.552 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2023-11-08T08:21:48.562 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.680+0000 7f41c779c700 1 Processor -- start 2023-11-08T08:21:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.680+0000 7f41c779c700 1 -- start start 2023-11-08T08:21:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c779c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41c006feb0 0x7f41c00cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c779c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c00b4b00 0x7f41c00cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c779c700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c00b4fc0 con 0x7f41c00ced70 2023-11-08T08:21:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c779c700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c0159c30 con 0x7f41c006feb0 2023-11-08T08:21:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c779c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c00b05e0 con 0x7f41c00b4b00 2023-11-08T08:21:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c4d37700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41c006feb0 0x7f41c00cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c5538700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f41c00ced70 0x7f41c00b43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39542/0 (socket says 172.21.15.116:39542) 2023-11-08T08:21:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.681+0000 7f41c5538700 1 -- 172.21.15.116:0/1099436833 learned_addr learned my addr 172.21.15.116:0/1099436833 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41c5d39700 1 --2- 172.21.15.116:0/1099436833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c00b4b00 0x7f41c00cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41c4d37700 1 -- 172.21.15.116:0/1099436833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c00b4b00 msgr2=0x7f41c00cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41c4d37700 1 --2- 172.21.15.116:0/1099436833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c00b4b00 0x7f41c00cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41c4d37700 1 -- 172.21.15.116:0/1099436833 >> v1:172.21.15.116:6789/0 conn(0x7f41c00ced70 legacy=0x7f41c00b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41b7fff700 1 -- 172.21.15.116:0/1099436833 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064330491 0 0) 0x7f41c00b4fc0 con 0x7f41c00ced70 2023-11-08T08:21:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41c4d37700 1 -- 172.21.15.116:0/1099436833 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c00b08c0 con 0x7f41c006feb0 2023-11-08T08:21:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41c4d37700 1 --2- 172.21.15.116:0/1099436833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c006feb0 0x7f41c00cd1d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009840 tx=0x7f41bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd53e191a6b0c031 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41b7fff700 1 -- 172.21.15.116:0/1099436833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc011830 con 0x7f41c006feb0 2023-11-08T08:21:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41b7fff700 1 -- 172.21.15.116:0/1099436833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc011990 con 0x7f41c006feb0 2023-11-08T08:21:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.682+0000 7f41b7fff700 1 -- 172.21.15.116:0/1099436833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc011d20 con 0x7f41c006feb0 2023-11-08T08:21:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 -- 172.21.15.116:0/1099436833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c006feb0 msgr2=0x7f41c00cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 --2- 172.21.15.116:0/1099436833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c006feb0 0x7f41c00cd1d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009840 tx=0x7f41bc009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 -- 172.21.15.116:0/1099436833 shutdown_connections 2023-11-08T08:21:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 --2- 172.21.15.116:0/1099436833 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c00b4b00 0x7f41c00cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 --2- 172.21.15.116:0/1099436833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c006feb0 0x7f41c00cd1d0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 -- 172.21.15.116:0/1099436833 >> 172.21.15.116:0/1099436833 conn(0x7f41c00b2fc0 msgr2=0x7f41c0071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 -- 172.21.15.116:0/1099436833 shutdown_connections 2023-11-08T08:21:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.683+0000 7f41c779c700 1 -- 172.21.15.116:0/1099436833 wait complete. 2023-11-08T08:21:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.684+0000 7f41c779c700 1 Processor -- start 2023-11-08T08:21:48.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.684+0000 7f41c779c700 1 -- start start 2023-11-08T08:21:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.684+0000 7f41c779c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c006feb0 0x7f41c00cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41c00b4b00 0x7f41c01575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 0x7f41c00c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0158170 con 0x7f41c00ced70 2023-11-08T08:21:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c00cb690 con 0x7f41c00b4b00 2023-11-08T08:21:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c00cb970 con 0x7f41c006feb0 2023-11-08T08:21:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 0x7f41c00c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 0x7f41c00c3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46182/0 (socket says 172.21.15.116:46182) 2023-11-08T08:21:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 -- 172.21.15.116:0/1356359356 learned_addr learned my addr 172.21.15.116:0/1356359356 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:48.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c4d37700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41c00b4b00 0x7f41c01575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5538700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c006feb0 0x7f41c00cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 -- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c006feb0 msgr2=0x7f41c00cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c006feb0 0x7f41c00cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 -- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41c00b4b00 msgr2=0x7f41c01575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41c00b4b00 0x7f41c01575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 -- 172.21.15.116:0/1356359356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0094a0 con 0x7f41c00ced70 2023-11-08T08:21:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c4d37700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41c00b4b00 0x7f41c01575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c5d39700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 0x7f41c00c3f40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f41b8007e40 tx=0x7f41b800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b800eea0 con 0x7f41c00ced70 2023-11-08T08:21:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b8014420 con 0x7f41c00ced70 2023-11-08T08:21:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c00b5260 con 0x7f41c00ced70 2023-11-08T08:21:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b80147b0 con 0x7f41c00ced70 2023-11-08T08:21:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.685+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c00b0ba0 con 0x7f41c00ced70 2023-11-08T08:21:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.686+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c006b280 con 0x7f41c00ced70 2023-11-08T08:21:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.687+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f41b8012050 con 0x7f41c00ced70 2023-11-08T08:21:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.687+0000 7f41b5ffb700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41a40682f0 0x7f41a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.687+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f41b8086360 con 0x7f41c00ced70 2023-11-08T08:21:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.687+0000 7f41c5538700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41a40682f0 0x7f41a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.687+0000 7f41c5538700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41a40682f0 0x7f41a406a7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000f60 tx=0x7f41b0007c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.691+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41b8052bc0 con 0x7f41c00ced70 2023-11-08T08:21:48.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.871+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41c0067370 con 0x7f41c00ced70 2023-11-08T08:21:48.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.872+0000 7f41b5ffb700 1 -- 172.21.15.116:0/1356359356 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17959 (secure 0 0 0) 0x7f41b80561e0 con 0x7f41c00ced70 2023-11-08T08:21:48.872 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:48.873 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:33.084884+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-11-08T08:21:29.779880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-11-08T08:21:31.046244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:48.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41a40682f0 msgr2=0x7f41a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41a40682f0 0x7f41a406a7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000f60 tx=0x7f41b0007c90 comp rx=0 tx=0).stop 2023-11-08T08:21:48.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 msgr2=0x7f41c00c3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:48.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 0x7f41c00c3f40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f41b8007e40 tx=0x7f41b800e3c0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 shutdown_connections 2023-11-08T08:21:48.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c00ced70 0x7f41c00c3f40 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f41c006feb0 0x7f41c00cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f41a40682f0 0x7f41a406a7a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 --2- 172.21.15.116:0/1356359356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41c00b4b00 0x7f41c01575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:48.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 >> 172.21.15.116:0/1356359356 conn(0x7f41c00b2fc0 msgr2=0x7f41c00cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:48.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 shutdown_connections 2023-11-08T08:21:48.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:48.874+0000 7f41c779c700 1 -- 172.21.15.116:0/1356359356 wait complete. 2023-11-08T08:21:48.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-08T08:21:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.007+0000 7f52c8581700 1 Processor -- start 2023-11-08T08:21:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.007+0000 7f52c8581700 1 -- start start 2023-11-08T08:21:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c8581700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52c01599b0 0x7f52c015d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c8581700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c0159ec0 con 0x7f52c00af0a0 2023-11-08T08:21:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c8581700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c015aa60 con 0x7f52c00b1650 2023-11-08T08:21:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c8581700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c015d8a0 con 0x7f52c01599b0 2023-11-08T08:21:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c5b1c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52c01599b0 0x7f52c015d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c6b1e700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f52c00af0a0 0x7f52c015a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39554/0 (socket says 172.21.15.116:39554) 2023-11-08T08:21:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c6b1e700 1 -- 172.21.15.116:0/1571991732 learned_addr learned my addr 172.21.15.116:0/1571991732 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c5b1c700 1 -- 172.21.15.116:0/1571991732 >> v1:172.21.15.129:6789/0 conn(0x7f52c00b1650 legacy=0x7f52c00b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c5b1c700 1 -- 172.21.15.116:0/1571991732 >> v1:172.21.15.116:6789/0 conn(0x7f52c00af0a0 legacy=0x7f52c015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c5b1c700 1 -- 172.21.15.116:0/1571991732 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c015db80 con 0x7f52c01599b0 2023-11-08T08:21:49.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c531b700 1 -- 172.21.15.116:0/1571991732 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325872278 0 0) 0x7f52c0159ec0 con 0x7f52c00af0a0 2023-11-08T08:21:49.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c5b1c700 1 --2- 172.21.15.116:0/1571991732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c01599b0 0x7f52c015d050 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009840 tx=0x7f52b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdf37dc701c5d95 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:49.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c531b700 1 -- 172.21.15.116:0/1571991732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0004230 con 0x7f52c01599b0 2023-11-08T08:21:49.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c531b700 1 -- 172.21.15.116:0/1571991732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0004390 con 0x7f52c01599b0 2023-11-08T08:21:49.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.008+0000 7f52c531b700 1 -- 172.21.15.116:0/1571991732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00036a0 con 0x7f52c01599b0 2023-11-08T08:21:49.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 -- 172.21.15.116:0/1571991732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c01599b0 msgr2=0x7f52c015d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 --2- 172.21.15.116:0/1571991732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c01599b0 0x7f52c015d050 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009840 tx=0x7f52b0009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:49.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 -- 172.21.15.116:0/1571991732 shutdown_connections 2023-11-08T08:21:49.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 --2- 172.21.15.116:0/1571991732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c01599b0 0x7f52c015d050 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 -- 172.21.15.116:0/1571991732 >> 172.21.15.116:0/1571991732 conn(0x7f52c006c010 msgr2=0x7f52c00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 -- 172.21.15.116:0/1571991732 shutdown_connections 2023-11-08T08:21:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.009+0000 7f52c8581700 1 -- 172.21.15.116:0/1571991732 wait complete. 2023-11-08T08:21:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.010+0000 7f52c8581700 1 Processor -- start 2023-11-08T08:21:49.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.010+0000 7f52c8581700 1 -- start start 2023-11-08T08:21:49.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c8581700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 0x7f52c0158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c8581700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52c00b1650 0x7f52c00c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c8581700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c00c5f30 0x7f52c01e3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c8581700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00c63d0 con 0x7f52c00af0a0 2023-11-08T08:21:49.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c8581700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01ef9f0 con 0x7f52c00c5f30 2023-11-08T08:21:49.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c8581700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01e4590 con 0x7f52c00b1650 2023-11-08T08:21:49.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 0x7f52c0158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 0x7f52c0158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46204/0 (socket says 172.21.15.116:46204) 2023-11-08T08:21:49.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 -- 172.21.15.116:0/3413930144 learned_addr learned my addr 172.21.15.116:0/3413930144 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:49.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c5b1c700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52c00b1650 0x7f52c00c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c6b1e700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c00c5f30 0x7f52c01e3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 -- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52c00b1650 msgr2=0x7f52c00c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52c00b1650 0x7f52c00c59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 -- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c00c5f30 msgr2=0x7f52c01e3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c00c5f30 0x7f52c01e3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c631d700 1 -- 172.21.15.116:0/3413930144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b00094a0 con 0x7f52c00af0a0 2023-11-08T08:21:49.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.011+0000 7f52c6b1e700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c00c5f30 0x7f52c01e3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:49.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.012+0000 7f52c631d700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 0x7f52c0158640 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00c4d0 tx=0x7f52bc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.012+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc011790 con 0x7f52c00af0a0 2023-11-08T08:21:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.012+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc0118f0 con 0x7f52c00af0a0 2023-11-08T08:21:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.012+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c015d590 con 0x7f52c00af0a0 2023-11-08T08:21:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.012+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc011ca0 con 0x7f52c00af0a0 2023-11-08T08:21:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.012+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c015de60 con 0x7f52c00af0a0 2023-11-08T08:21:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.013+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c00743f0 con 0x7f52c00af0a0 2023-11-08T08:21:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.013+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f52bc010050 con 0x7f52c00af0a0 2023-11-08T08:21:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.013+0000 7f52b6ffd700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52a40682f0 0x7f52a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.014+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f52bc0858b0 con 0x7f52c00af0a0 2023-11-08T08:21:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.014+0000 7f52c5b1c700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52a40682f0 0x7f52a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.014+0000 7f52c5b1c700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52a40682f0 0x7f52a406a7a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009840 tx=0x7f52b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:49.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.018+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52bc052110 con 0x7f52c00af0a0 2023-11-08T08:21:49.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:21:49.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:21:49.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.244+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f52c006cd10 con 0x7f52c00af0a0 2023-11-08T08:21:49.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-08T08:21:49.525+0000 7f50f4469700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-08T08:21:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.536+0000 7f52b6ffd700 1 -- 172.21.15.116:0/3413930144 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f52bc055730 con 0x7f52c00af0a0 2023-11-08T08:21:49.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.538+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52a40682f0 msgr2=0x7f52a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.538+0000 7f52c8581700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52a40682f0 0x7f52a406a7a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009840 tx=0x7f52b0008030 comp rx=0 tx=0).stop 2023-11-08T08:21:49.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.538+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 msgr2=0x7f52c0158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.538+0000 7f52c8581700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 0x7f52c0158640 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00c4d0 tx=0x7f52bc00c7e0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 shutdown_connections 2023-11-08T08:21:49.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f52a40682f0 0x7f52a406a7a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c00c5f30 0x7f52c01e3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c00af0a0 0x7f52c0158640 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 --2- 172.21.15.116:0/3413930144 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f52c00b1650 0x7f52c00c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 >> 172.21.15.116:0/3413930144 conn(0x7f52c006c010 msgr2=0x7f52c00b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:49.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 shutdown_connections 2023-11-08T08:21:49.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.539+0000 7f52c8581700 1 -- 172.21.15.116:0/3413930144 wait complete. 2023-11-08T08:21:49.542 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-08T08:21:49.542 INFO:tasks.ceph.mds.d.smithi129.stderr: -3186> 2023-11-08T08:21:37.570+0000 7f28daf31700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-08T08:21:49.542 INFO:tasks.ceph.mds.d.smithi129.stderr: -1682> 2023-11-08T08:21:41.501+0000 7f28d772a700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6366=0-4194304@1} caps={6366=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6366 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x556ba6acf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-08T08:21:49.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-08T08:21:49.570 INFO:tasks.ceph.mds.d.smithi129.stdout:starting mds.d at 2023-11-08T08:21:49.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.713+0000 7f71dd422700 1 Processor -- start 2023-11-08T08:21:49.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.713+0000 7f71dd422700 1 -- start start 2023-11-08T08:21:49.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71dd422700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f71d815fe50 0x7f71d806d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71dd422700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71d80b3b80 0x7f71d8160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71dd422700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d8161370 con 0x7f71d80b27d0 2023-11-08T08:21:49.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71dd422700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d8160220 con 0x7f71d80b3b80 2023-11-08T08:21:49.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71dd422700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d80cf700 con 0x7f71d815fe50 2023-11-08T08:21:49.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71d80b3b80 0x7f71d8160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71d80b3b80 0x7f71d8160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46210/0 (socket says 172.21.15.116:46210) 2023-11-08T08:21:49.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 -- 172.21.15.116:0/1954243869 learned_addr learned my addr 172.21.15.116:0/1954243869 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:49.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d6ffd700 1 --2- 172.21.15.116:0/1954243869 >> v2:172.21.15.129:3300/0 conn(0x7f71d815fe50 0x7f71d806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 -- 172.21.15.116:0/1954243869 >> v2:172.21.15.129:3300/0 conn(0x7f71d815fe50 msgr2=0x7f71d806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 --2- 172.21.15.116:0/1954243869 >> v2:172.21.15.129:3300/0 conn(0x7f71d815fe50 0x7f71d806d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 -- 172.21.15.116:0/1954243869 >> v1:172.21.15.129:6789/0 conn(0x7f71d80b27d0 legacy=0x7f71d80cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 -- 172.21.15.116:0/1954243869 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80b3fd0 con 0x7f71d80b3b80 2023-11-08T08:21:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d67fc700 1 --2- 172.21.15.116:0/1954243869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d80b3b80 0x7f71d8160ad0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f71d80d1e30 tx=0x7f71cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d75879ea85ff0d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:49.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d5ffb700 1 -- 172.21.15.116:0/1954243869 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc011830 con 0x7f71d80b3b80 2023-11-08T08:21:49.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d5ffb700 1 -- 172.21.15.116:0/1954243869 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc011990 con 0x7f71d80b3b80 2023-11-08T08:21:49.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.714+0000 7f71d5ffb700 1 -- 172.21.15.116:0/1954243869 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc011d20 con 0x7f71d80b3b80 2023-11-08T08:21:49.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 -- 172.21.15.116:0/1954243869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d80b3b80 msgr2=0x7f71d8160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 --2- 172.21.15.116:0/1954243869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d80b3b80 0x7f71d8160ad0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f71d80d1e30 tx=0x7f71cc009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 -- 172.21.15.116:0/1954243869 shutdown_connections 2023-11-08T08:21:49.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 --2- 172.21.15.116:0/1954243869 >> v2:172.21.15.129:3300/0 conn(0x7f71d815fe50 0x7f71d806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 --2- 172.21.15.116:0/1954243869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d80b3b80 0x7f71d8160ad0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 -- 172.21.15.116:0/1954243869 >> 172.21.15.116:0/1954243869 conn(0x7f71d806ea80 msgr2=0x7f71d806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:49.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 -- 172.21.15.116:0/1954243869 shutdown_connections 2023-11-08T08:21:49.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.715+0000 7f71dd422700 1 -- 172.21.15.116:0/1954243869 wait complete. 2023-11-08T08:21:49.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.717+0000 7f71dd422700 1 Processor -- start 2023-11-08T08:21:49.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.717+0000 7f71dd422700 1 -- start start 2023-11-08T08:21:49.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.717+0000 7f71dd422700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80b27d0 0x7f71d8157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.717+0000 7f71dd422700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71d80b3b80 0x7f71d80cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71dd422700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 0x7f71d80c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71dd422700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80cc970 con 0x7f71d815fe50 2023-11-08T08:21:49.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71dd422700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80cbcf0 con 0x7f71d80b27d0 2023-11-08T08:21:49.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71dd422700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80cbfd0 con 0x7f71d80b3b80 2023-11-08T08:21:49.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 0x7f71d80c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 0x7f71d80c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46218/0 (socket says 172.21.15.116:46218) 2023-11-08T08:21:49.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 -- 172.21.15.116:0/1658292600 learned_addr learned my addr 172.21.15.116:0/1658292600 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:49.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d67fc700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71d80b3b80 0x7f71d80cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d6ffd700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80b27d0 0x7f71d8157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 -- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71d80b3b80 msgr2=0x7f71d80cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71d80b3b80 0x7f71d80cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 -- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80b27d0 msgr2=0x7f71d8157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:49.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80b27d0 0x7f71d8157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:49.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 -- 172.21.15.116:0/1658292600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc009550 con 0x7f71d815fe50 2023-11-08T08:21:49.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71d77fe700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 0x7f71d80c9e10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f71d000cb50 tx=0x7f71d000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:49.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0003c00 con 0x7f71d815fe50 2023-11-08T08:21:49.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d0003d60 con 0x7f71d815fe50 2023-11-08T08:21:49.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d80cf0b0 con 0x7f71d815fe50 2023-11-08T08:21:49.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d000ab70 con 0x7f71d815fe50 2023-11-08T08:21:49.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.718+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d80cedd0 con 0x7f71d815fe50 2023-11-08T08:21:49.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.719+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d80cf390 con 0x7f71d815fe50 2023-11-08T08:21:49.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.720+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f71d0010050 con 0x7f71d815fe50 2023-11-08T08:21:49.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.720+0000 7f71bf7fe700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71c40682f0 0x7f71c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:49.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.720+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f71d0085bd0 con 0x7f71d815fe50 2023-11-08T08:21:49.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.720+0000 7f71d6ffd700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71c40682f0 0x7f71c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:49.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.720+0000 7f71d6ffd700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71c40682f0 0x7f71c406a7a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009b70 tx=0x7f71c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:49.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.724+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f71d00523d0 con 0x7f71d815fe50 2023-11-08T08:21:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:49.948+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f71d80ad940 con 0x7f71d815fe50 2023-11-08T08:21:50.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.538+0000 7f71bf7fe700 1 -- 172.21.15.116:0/1658292600 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f71d80ad940 con 0x7f71d815fe50 2023-11-08T08:21:50.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71c40682f0 msgr2=0x7f71c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:50.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71c40682f0 0x7f71c406a7a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009b70 tx=0x7f71c8008040 comp rx=0 tx=0).stop 2023-11-08T08:21:50.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 msgr2=0x7f71d80c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:50.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 0x7f71d80c9e10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f71d000cb50 tx=0x7f71d000cf10 comp rx=0 tx=0).stop 2023-11-08T08:21:50.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 shutdown_connections 2023-11-08T08:21:50.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d815fe50 0x7f71d80c9e10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f71c40682f0 0x7f71c406a7a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80b27d0 0x7f71d8157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 --2- 172.21.15.116:0/1658292600 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f71d80b3b80 0x7f71d80cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 >> 172.21.15.116:0/1658292600 conn(0x7f71d806ea80 msgr2=0x7f71d80b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:50.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 shutdown_connections 2023-11-08T08:21:50.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.540+0000 7f71dd422700 1 -- 172.21.15.116:0/1658292600 wait complete. 2023-11-08T08:21:50.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-08T08:21:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.683+0000 7fb8748e1700 1 Processor -- start 2023-11-08T08:21:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.683+0000 7fb8748e1700 1 -- start start 2023-11-08T08:21:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb8748e1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87006ba90 0x7fb87015a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb8748e1700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8700ceea0 0x7fb87015a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb8748e1700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700cea10 con 0x7fb87015e070 2023-11-08T08:21:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb8748e1700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87015ae80 con 0x7fb8700ceea0 2023-11-08T08:21:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb8748e1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87015abd0 con 0x7fb87006ba90 2023-11-08T08:21:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86ed9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8700ceea0 0x7fb87015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86e59c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb87015e070 0x7fb87006b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39558/0 (socket says 172.21.15.116:39558) 2023-11-08T08:21:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86e59c700 1 -- 172.21.15.116:0/1532948984 learned_addr learned my addr 172.21.15.116:0/1532948984 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86ed9d700 1 -- 172.21.15.116:0/1532948984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87006ba90 msgr2=0x7fb87015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86ed9d700 1 --2- 172.21.15.116:0/1532948984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87006ba90 0x7fb87015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86ed9d700 1 -- 172.21.15.116:0/1532948984 >> v1:172.21.15.116:6789/0 conn(0x7fb87015e070 legacy=0x7fb87006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86ed9d700 1 -- 172.21.15.116:0/1532948984 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8700cbd10 con 0x7fb8700ceea0 2023-11-08T08:21:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.684+0000 7fb86d59a700 1 -- 172.21.15.116:0/1532948984 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887823121 0 0) 0x7fb8700cea10 con 0x7fb87015e070 2023-11-08T08:21:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb86ed9d700 1 --2- 172.21.15.116:0/1532948984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87015a690 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb868009d90 tx=0x7fb86800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1af9ad138fe76dd server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb86d59a700 1 -- 172.21.15.116:0/1532948984 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86800e070 con 0x7fb8700ceea0 2023-11-08T08:21:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb86d59a700 1 -- 172.21.15.116:0/1532948984 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86800bbd0 con 0x7fb8700ceea0 2023-11-08T08:21:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb86d59a700 1 -- 172.21.15.116:0/1532948984 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868012430 con 0x7fb8700ceea0 2023-11-08T08:21:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 -- 172.21.15.116:0/1532948984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 msgr2=0x7fb87015a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 --2- 172.21.15.116:0/1532948984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87015a690 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb868009d90 tx=0x7fb86800b330 comp rx=0 tx=0).stop 2023-11-08T08:21:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 -- 172.21.15.116:0/1532948984 shutdown_connections 2023-11-08T08:21:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 --2- 172.21.15.116:0/1532948984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87015a690 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 --2- 172.21.15.116:0/1532948984 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87006ba90 0x7fb87015a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 -- 172.21.15.116:0/1532948984 >> 172.21.15.116:0/1532948984 conn(0x7fb8700cfc30 msgr2=0x7fb8700d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:50.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 -- 172.21.15.116:0/1532948984 shutdown_connections 2023-11-08T08:21:50.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.685+0000 7fb8748e1700 1 -- 172.21.15.116:0/1532948984 wait complete. 2023-11-08T08:21:50.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 Processor -- start 2023-11-08T08:21:50.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 -- start start 2023-11-08T08:21:50.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb87006ba90 0x7fb8701f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87014aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87015e070 0x7fb8700c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87006d2d0 con 0x7fb8700ceea0 2023-11-08T08:21:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87006cca0 con 0x7fb87006ba90 2023-11-08T08:21:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.687+0000 7fb8748e1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8701f6db0 con 0x7fb87015e070 2023-11-08T08:21:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87014aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87014aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46244/0 (socket says 172.21.15.116:46244) 2023-11-08T08:21:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 -- 172.21.15.116:0/875777371 learned_addr learned my addr 172.21.15.116:0/875777371 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86ed9d700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87015e070 0x7fb8700c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86e59c700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb87006ba90 0x7fb8701f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 -- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87015e070 msgr2=0x7fb8700c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87015e070 0x7fb8700c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 -- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb87006ba90 msgr2=0x7fb8701f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb87006ba90 0x7fb8701f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 -- 172.21.15.116:0/875777371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868009a70 con 0x7fb8700ceea0 2023-11-08T08:21:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86ed9d700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87015e070 0x7fb8700c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb86dd9b700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87014aba0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb86000b450 tx=0x7fb86000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860011830 con 0x7fb8700ceea0 2023-11-08T08:21:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb860011990 con 0x7fb8700ceea0 2023-11-08T08:21:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8700ce3c0 con 0x7fb8700ceea0 2023-11-08T08:21:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860011d40 con 0x7fb8700ceea0 2023-11-08T08:21:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.688+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8700ce0e0 con 0x7fb8700ceea0 2023-11-08T08:21:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.689+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700ce6a0 con 0x7fb8700ceea0 2023-11-08T08:21:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.690+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb860021070 con 0x7fb8700ceea0 2023-11-08T08:21:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.690+0000 7fb85effd700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb8500682f0 0x7fb85006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.690+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb860085bb0 con 0x7fb8700ceea0 2023-11-08T08:21:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.690+0000 7fb86e59c700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb8500682f0 0x7fb85006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.690+0000 7fb86e59c700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb8500682f0 0x7fb85006a7a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb864000f60 tx=0x7fb864007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.694+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb86008c080 con 0x7fb8700ceea0 2023-11-08T08:21:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:50.873+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb87015bd10 con 0x7fb8700ceea0 2023-11-08T08:21:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.594+0000 7fb85effd700 1 -- 172.21.15.116:0/875777371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb8600150d0 con 0x7fb8700ceea0 2023-11-08T08:21:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb8500682f0 msgr2=0x7fb85006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb8500682f0 0x7fb85006a7a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb864000f60 tx=0x7fb864007be0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 msgr2=0x7fb87014aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87014aba0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb86000b450 tx=0x7fb86000b810 comp rx=0 tx=0).stop 2023-11-08T08:21:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 shutdown_connections 2023-11-08T08:21:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb87015e070 0x7fb8700c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb8500682f0 0x7fb85006a7a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb87006ba90 0x7fb8701f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 --2- 172.21.15.116:0/875777371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8700ceea0 0x7fb87014aba0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 >> 172.21.15.116:0/875777371 conn(0x7fb8700cfc30 msgr2=0x7fb8700d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 shutdown_connections 2023-11-08T08:21:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.597+0000 7fb8748e1700 1 -- 172.21.15.116:0/875777371 wait complete. 2023-11-08T08:21:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2023-11-08T08:21:51.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-08T08:21:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.743+0000 7f17d073f700 1 Processor -- start 2023-11-08T08:21:51.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.743+0000 7f17d073f700 1 -- start start 2023-11-08T08:21:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17d073f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 0x7f17c80b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17d073f700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17c80b22e0 0x7f17c80d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17d073f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80b32b0 con 0x7f17c80b1c90 2023-11-08T08:21:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17d073f700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c80b26e0 con 0x7f17c80b22e0 2023-11-08T08:21:51.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17d073f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80b2980 con 0x7f17c815cce0 2023-11-08T08:21:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17cecdc700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17c80b1c90 0x7f17c80afd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39564/0 (socket says 172.21.15.116:39564) 2023-11-08T08:21:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.744+0000 7f17cecdc700 1 -- 172.21.15.116:0/366259785 learned_addr learned my addr 172.21.15.116:0/366259785 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17ce4db700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 0x7f17c80b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 --2- 172.21.15.116:0/366259785 >> v2:172.21.15.129:3300/0 conn(0x7f17c80b22e0 0x7f17c80d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969197843 0 0) 0x7f17c80b32b0 con 0x7f17c80b1c90 2023-11-08T08:21:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b00035b0 con 0x7f17c80b1c90 2023-11-08T08:21:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196663884 0 0) 0x7f17b00035b0 con 0x7f17c80b1c90 2023-11-08T08:21:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 msgr2=0x7f17c80b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 0x7f17c80b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 >> v2:172.21.15.129:3300/0 conn(0x7f17c80b22e0 msgr2=0x7f17c80d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 --2- 172.21.15.116:0/366259785 >> v2:172.21.15.129:3300/0 conn(0x7f17c80b22e0 0x7f17c80d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80b2c60 con 0x7f17c80b1c90 2023-11-08T08:21:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f17b8001670 con 0x7f17c80b1c90 2023-11-08T08:21:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 >> v1:172.21.15.116:6789/0 conn(0x7f17c80b1c90 legacy=0x7f17c80afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0003a00 0x7f17b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b0004430 0x7f17b00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 0x7f17b0004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80b26e0 con 0x7f17b0004430 2023-11-08T08:21:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80b2980 con 0x7f17b0003a00 2023-11-08T08:21:51.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b000ba10 con 0x7f17c815cce0 2023-11-08T08:21:51.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b0004430 0x7f17b00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17ce4db700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0003a00 0x7f17b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cecdc700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 0x7f17b0004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 -- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 msgr2=0x7f17b0004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:51.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c815cce0 0x7f17b0004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 -- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0003a00 msgr2=0x7f17b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:51.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0003a00 0x7f17b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 -- 172.21.15.116:0/366259785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b00035b0 con 0x7f17b0004430 2023-11-08T08:21:51.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cdcda700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b0004430 0x7f17b00048f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009b60 tx=0x7f17b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2926b338b2e66557 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:51.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.745+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00e6c0 con 0x7f17b0004430 2023-11-08T08:21:51.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc00e820 con 0x7f17b0004430 2023-11-08T08:21:51.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17cd4d9700 1 -- 172.21.15.116:0/366259785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00eb90 con 0x7f17b0004430 2023-11-08T08:21:51.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17d073f700 1 -- 172.21.15.116:0/366259785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b0004430 msgr2=0x7f17b00048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:51.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17d073f700 1 --2- 172.21.15.116:0/366259785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b0004430 0x7f17b00048f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009b60 tx=0x7f17b0005880 comp rx=0 tx=0).stop 2023-11-08T08:21:51.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17cecdc700 1 -- 172.21.15.116:0/366259785 reap_dead start 2023-11-08T08:21:51.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17d073f700 1 -- 172.21.15.116:0/366259785 shutdown_connections 2023-11-08T08:21:51.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17d073f700 1 -- 172.21.15.116:0/366259785 >> 172.21.15.116:0/366259785 conn(0x7f17c809c4d0 msgr2=0x7f17c80d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:51.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17d073f700 1 -- 172.21.15.116:0/366259785 shutdown_connections 2023-11-08T08:21:51.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.746+0000 7f17d073f700 1 -- 172.21.15.116:0/366259785 wait complete. 2023-11-08T08:21:51.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.747+0000 7f17d073f700 1 Processor -- start 2023-11-08T08:21:51.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.747+0000 7f17d073f700 1 -- start start 2023-11-08T08:21:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17d073f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0004430 0x7f17c81596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17d073f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c80a0780 0x7f17c806be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17d073f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 0x7f17c806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17d073f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c809f790 con 0x7f17c80a0bf0 2023-11-08T08:21:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17d073f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80a02f0 con 0x7f17b0004430 2023-11-08T08:21:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17d073f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8159bf0 con 0x7f17c80a0780 2023-11-08T08:21:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 0x7f17c806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 0x7f17c806c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46254/0 (socket says 172.21.15.116:46254) 2023-11-08T08:21:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 -- 172.21.15.116:0/2879341693 learned_addr learned my addr 172.21.15.116:0/2879341693 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cdcda700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c80a0780 0x7f17c806be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17ce4db700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0004430 0x7f17c81596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 -- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c80a0780 msgr2=0x7f17c806be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c80a0780 0x7f17c806be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 -- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0004430 msgr2=0x7f17c81596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0004430 0x7f17c81596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cecdc700 1 -- 172.21.15.116:0/2879341693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80b4750 con 0x7f17c80a0bf0 2023-11-08T08:21:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17cdcda700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c80a0780 0x7f17c806be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.748+0000 7f17ce4db700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0004430 0x7f17c81596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.749+0000 7f17cecdc700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 0x7f17c806c3a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f17b8000fb0 tx=0x7f17b800ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.749+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8010510 con 0x7f17c80a0bf0 2023-11-08T08:21:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.749+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b8010690 con 0x7f17c80a0bf0 2023-11-08T08:21:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.749+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8010a40 con 0x7f17c80a0bf0 2023-11-08T08:21:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.749+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c80b2f40 con 0x7f17c80a0bf0 2023-11-08T08:21:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.749+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c80b48e0 con 0x7f17c80a0bf0 2023-11-08T08:21:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.750+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80743f0 con 0x7f17c80a0bf0 2023-11-08T08:21:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.750+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f17b8010ba0 con 0x7f17c80a0bf0 2023-11-08T08:21:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.751+0000 7f17c6ffd700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b40682f0 0x7f17b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.751+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5710+0+0 (secure 0 0 0) 0x7f17b80850b0 con 0x7f17c80a0bf0 2023-11-08T08:21:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.751+0000 7f17ce4db700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b40682f0 0x7f17b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.751+0000 7f17ce4db700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b40682f0 0x7f17b406a7a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17c806cb40 tx=0x7f17c000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.755+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17b8051980 con 0x7f17c80a0bf0 2023-11-08T08:21:51.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:51.938+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17c80a0430 con 0x7f17c80a0bf0 2023-11-08T08:21:52.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.604+0000 7f17c6ffd700 1 -- 172.21.15.116:0/2879341693 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7f17b8019020 con 0x7f17c80a0bf0 2023-11-08T08:21:52.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b40682f0 msgr2=0x7f17b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b40682f0 0x7f17b406a7a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17c806cb40 tx=0x7f17c000b040 comp rx=0 tx=0).stop 2023-11-08T08:21:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 msgr2=0x7f17c806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 0x7f17c806c3a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f17b8000fb0 tx=0x7f17b800ea90 comp rx=0 tx=0).stop 2023-11-08T08:21:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 shutdown_connections 2023-11-08T08:21:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c80a0bf0 0x7f17c806c3a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b40682f0 0x7f17b406a7a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b0004430 0x7f17c81596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 --2- 172.21.15.116:0/2879341693 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17c80a0780 0x7f17c806be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 >> 172.21.15.116:0/2879341693 conn(0x7f17c809c4d0 msgr2=0x7f17c80afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.607+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 shutdown_connections 2023-11-08T08:21:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.608+0000 7f17d073f700 1 -- 172.21.15.116:0/2879341693 wait complete. 2023-11-08T08:21:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2023-11-08T08:21:52.620 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-08T08:21:52.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.748+0000 7f74ae53a700 1 Processor -- start 2023-11-08T08:21:52.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.748+0000 7f74ae53a700 1 -- start start 2023-11-08T08:21:52.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74ae53a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f74a81599b0 0x7f74a815d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:52.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74ae53a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80af0a0 0x7f74a815a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:52.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74ae53a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a8159ec0 con 0x7f74a80b1650 2023-11-08T08:21:52.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74ae53a700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a815d8a0 con 0x7f74a81599b0 2023-11-08T08:21:52.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74ae53a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a815db80 con 0x7f74a80af0a0 2023-11-08T08:21:52.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f74a81599b0 0x7f74a815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f74a81599b0 0x7f74a815d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53764/0 (socket says 172.21.15.116:53764) 2023-11-08T08:21:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 -- 172.21.15.116:0/589835781 learned_addr learned my addr 172.21.15.116:0/589835781 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74acad7700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80af0a0 0x7f74a815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 -- 172.21.15.116:0/589835781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80af0a0 msgr2=0x7f74a815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:52.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80af0a0 0x7f74a815a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 -- 172.21.15.116:0/589835781 >> v1:172.21.15.129:6789/0 conn(0x7f74a80b1650 legacy=0x7f74a80b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 -- 172.21.15.116:0/589835781 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a815de60 con 0x7f74a81599b0 2023-11-08T08:21:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74acad7700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80af0a0 0x7f74a815a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:52.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a77fe700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a81599b0 0x7f74a815d050 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f749800bb90 tx=0x7f749800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e691f4d35682629 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a6ffd700 1 -- 172.21.15.116:0/589835781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749800dd10 con 0x7f74a81599b0 2023-11-08T08:21:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.749+0000 7f74a6ffd700 1 -- 172.21.15.116:0/589835781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749800de70 con 0x7f74a81599b0 2023-11-08T08:21:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74a6ffd700 1 -- 172.21.15.116:0/589835781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498004230 con 0x7f74a81599b0 2023-11-08T08:21:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 -- 172.21.15.116:0/589835781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a81599b0 msgr2=0x7f74a815d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a81599b0 0x7f74a815d050 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f749800bb90 tx=0x7f749800bea0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 -- 172.21.15.116:0/589835781 shutdown_connections 2023-11-08T08:21:52.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80af0a0 0x7f74a815a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 --2- 172.21.15.116:0/589835781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a81599b0 0x7f74a815d050 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 -- 172.21.15.116:0/589835781 >> 172.21.15.116:0/589835781 conn(0x7f74a806c010 msgr2=0x7f74a80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:52.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 -- 172.21.15.116:0/589835781 shutdown_connections 2023-11-08T08:21:52.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.750+0000 7f74ae53a700 1 -- 172.21.15.116:0/589835781 wait complete. 2023-11-08T08:21:52.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.751+0000 7f74ae53a700 1 Processor -- start 2023-11-08T08:21:52.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.751+0000 7f74ae53a700 1 -- start start 2023-11-08T08:21:52.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74ae53a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 0x7f74a80b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:52.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74ae53a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80b1650 0x7f74a8150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:52.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74ae53a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74a81514d0 0x7f74a80c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:52.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74ae53a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a8151970 con 0x7f74a80af0a0 2023-11-08T08:21:52.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74ae53a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80c5ae0 con 0x7f74a81514d0 2023-11-08T08:21:52.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74ae53a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80c4a40 con 0x7f74a80b1650 2023-11-08T08:21:52.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74a7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 0x7f74a80b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:52.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74a7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 0x7f74a80b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53770/0 (socket says 172.21.15.116:53770) 2023-11-08T08:21:52.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74a7fff700 1 -- 172.21.15.116:0/1120545951 learned_addr learned my addr 172.21.15.116:0/1120545951 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:52.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74a77fe700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80b1650 0x7f74a8150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:52.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.752+0000 7f74acad7700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74a81514d0 0x7f74a80c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:52.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a7fff700 1 -- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80b1650 msgr2=0x7f74a8150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a7fff700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80b1650 0x7f74a8150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a7fff700 1 -- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74a81514d0 msgr2=0x7f74a80c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a7fff700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74a81514d0 0x7f74a80c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a7fff700 1 -- 172.21.15.116:0/1120545951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749800b7f0 con 0x7f74a80af0a0 2023-11-08T08:21:52.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a7fff700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 0x7f74a80b4e40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f749c00b4a0 tx=0x7f749c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:52.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c011830 con 0x7f74a80af0a0 2023-11-08T08:21:52.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c011990 con 0x7f74a80af0a0 2023-11-08T08:21:52.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a815a890 con 0x7f74a80af0a0 2023-11-08T08:21:52.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c011d40 con 0x7f74a80af0a0 2023-11-08T08:21:52.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.753+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a815d590 con 0x7f74a80af0a0 2023-11-08T08:21:52.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.754+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a815aa20 con 0x7f74a80af0a0 2023-11-08T08:21:52.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.754+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f749c00f050 con 0x7f74a80af0a0 2023-11-08T08:21:52.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.755+0000 7f74a4ff9700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f74940682f0 0x7f749406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:52.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.755+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f749c0855b0 con 0x7f74a80af0a0 2023-11-08T08:21:52.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.755+0000 7f74a77fe700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f74940682f0 0x7f749406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:52.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.755+0000 7f74a77fe700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f74940682f0 0x7f749406a7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f749800bb90 tx=0x7f7498013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:52.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.759+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f749c052110 con 0x7f74a80af0a0 2023-11-08T08:21:52.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.940+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f74a80743f0 con 0x7f74a80af0a0 2023-11-08T08:21:52.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.944+0000 7f74a4ff9700 1 -- 172.21.15.116:0/1120545951 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f749c055730 con 0x7f74a80af0a0 2023-11-08T08:21:52.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f74940682f0 msgr2=0x7f749406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f74940682f0 0x7f749406a7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f749800bb90 tx=0x7f7498013000 comp rx=0 tx=0).stop 2023-11-08T08:21:52.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 msgr2=0x7f74a80b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:52.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 0x7f74a80b4e40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f749c00b4a0 tx=0x7f749c00b860 comp rx=0 tx=0).stop 2023-11-08T08:21:52.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 shutdown_connections 2023-11-08T08:21:52.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f74940682f0 0x7f749406a7a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74a81514d0 0x7f74a80c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74a80af0a0 0x7f74a80b4e40 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 --2- 172.21.15.116:0/1120545951 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f74a80b1650 0x7f74a8150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:52.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 >> 172.21.15.116:0/1120545951 conn(0x7f74a806c010 msgr2=0x7f74a80aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:52.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 shutdown_connections 2023-11-08T08:21:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:52.946+0000 7f74ae53a700 1 -- 172.21.15.116:0/1120545951 wait complete. 2023-11-08T08:21:52.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2023-11-08T08:21:53.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.078+0000 7fbbb6371700 1 Processor -- start 2023-11-08T08:21:53.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.078+0000 7fbbb6371700 1 -- start start 2023-11-08T08:21:53.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb6371700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbb006a270 0x7fbbb006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb6371700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb0070d90 0x7fbbb00711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb6371700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbbb006da40 0x7fbbb006dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb6371700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb0151a50 con 0x7fbbb006a270 2023-11-08T08:21:53.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb6371700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb015b440 con 0x7fbbb006da40 2023-11-08T08:21:53.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb6371700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb006ab80 con 0x7fbbb0070d90 2023-11-08T08:21:53.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbb006a270 0x7fbbb006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbb006a270 0x7fbbb006a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53784/0 (socket says 172.21.15.116:53784) 2023-11-08T08:21:53.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 -- 172.21.15.116:0/3691749045 learned_addr learned my addr 172.21.15.116:0/3691749045 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:53.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaf7fe700 1 --2- 172.21.15.116:0/3691749045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb0070d90 0x7fbbb00711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbb490e700 1 --2- 172.21.15.116:0/3691749045 >> v2:172.21.15.129:3300/0 conn(0x7fbbb006da40 0x7fbbb006dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 -- 172.21.15.116:0/3691749045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb0070d90 msgr2=0x7fbbb00711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:53.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 --2- 172.21.15.116:0/3691749045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb0070d90 0x7fbbb00711e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 -- 172.21.15.116:0/3691749045 >> v2:172.21.15.129:3300/0 conn(0x7fbbb006da40 msgr2=0x7fbbb006dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:53.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 --2- 172.21.15.116:0/3691749045 >> v2:172.21.15.129:3300/0 conn(0x7fbbb006da40 0x7fbbb006dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 -- 172.21.15.116:0/3691749045 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0071ac0 con 0x7fbbb006a270 2023-11-08T08:21:53.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.079+0000 7fbbaffff700 1 --2- 172.21.15.116:0/3691749045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb006a270 0x7fbbb006a640 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00c6d50 tx=0x7fbba4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d5d86ed192bdc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:53.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbaeffd700 1 -- 172.21.15.116:0/3691749045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba400b870 con 0x7fbbb006a270 2023-11-08T08:21:53.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbaeffd700 1 -- 172.21.15.116:0/3691749045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba400b9d0 con 0x7fbbb006a270 2023-11-08T08:21:53.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbaeffd700 1 -- 172.21.15.116:0/3691749045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba400bd60 con 0x7fbbb006a270 2023-11-08T08:21:53.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 -- 172.21.15.116:0/3691749045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb006a270 msgr2=0x7fbbb006a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 --2- 172.21.15.116:0/3691749045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb006a270 0x7fbbb006a640 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00c6d50 tx=0x7fbba4009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 -- 172.21.15.116:0/3691749045 shutdown_connections 2023-11-08T08:21:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 --2- 172.21.15.116:0/3691749045 >> v2:172.21.15.129:3300/0 conn(0x7fbbb006da40 0x7fbbb006dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 --2- 172.21.15.116:0/3691749045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb006a270 0x7fbbb006a640 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 --2- 172.21.15.116:0/3691749045 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb0070d90 0x7fbbb00711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 -- 172.21.15.116:0/3691749045 >> 172.21.15.116:0/3691749045 conn(0x7fbbb006c010 msgr2=0x7fbbb00cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:53.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 -- 172.21.15.116:0/3691749045 shutdown_connections 2023-11-08T08:21:53.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.080+0000 7fbbb6371700 1 -- 172.21.15.116:0/3691749045 wait complete. 2023-11-08T08:21:53.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.081+0000 7fbbb6371700 1 Processor -- start 2023-11-08T08:21:53.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 -- start start 2023-11-08T08:21:53.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb006a270 0x7fbbb00cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb006da40 0x7fbbb00c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 0x7fbbb00c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00ceaf0 con 0x7fbbb0070d90 2023-11-08T08:21:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c5f10 con 0x7fbbb006da40 2023-11-08T08:21:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb6371700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb0159b30 con 0x7fbbb006a270 2023-11-08T08:21:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb490e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 0x7fbbb00c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb490e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 0x7fbbb00c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53796/0 (socket says 172.21.15.116:53796) 2023-11-08T08:21:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.082+0000 7fbbb490e700 1 -- 172.21.15.116:0/737908145 learned_addr learned my addr 172.21.15.116:0/737908145 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbaffff700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb006a270 0x7fbbb00cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbaf7fe700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb006da40 0x7fbbb00c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb490e700 1 -- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb006a270 msgr2=0x7fbbb00cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb490e700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb006a270 0x7fbbb00cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb490e700 1 -- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb006da40 msgr2=0x7fbbb00c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb490e700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb006da40 0x7fbbb00c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb490e700 1 -- 172.21.15.116:0/737908145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4009550 con 0x7fbbb0070d90 2023-11-08T08:21:53.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbaf7fe700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb006da40 0x7fbbb00c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:53.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb490e700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 0x7fbbb00c4950 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbba8007e60 tx=0x7fbba800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:53.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba800ede0 con 0x7fbbb0070d90 2023-11-08T08:21:53.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba8014420 con 0x7fbbb0070d90 2023-11-08T08:21:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb015b0d0 con 0x7fbbb0070d90 2023-11-08T08:21:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80147d0 con 0x7fbbb0070d90 2023-11-08T08:21:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.083+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb015adf0 con 0x7fbbb0070d90 2023-11-08T08:21:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.084+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb015aac0 con 0x7fbbb0070d90 2023-11-08T08:21:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.084+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbba8012050 con 0x7fbbb0070d90 2023-11-08T08:21:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.085+0000 7fbbacff9700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.085+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fbba8085f10 con 0x7fbbb0070d90 2023-11-08T08:21:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.085+0000 7fbbaffff700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.085+0000 7fbbaffff700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbba4000c00 tx=0x7fbba4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:53.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.089+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbba8052a70 con 0x7fbbb0070d90 2023-11-08T08:21:53.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.341+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7fbbb00743f0 con 0x7fbbb0070d90 2023-11-08T08:21:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.599+0000 7fbbacff9700 1 -- 172.21.15.116:0/737908145 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v180) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbba8056090 con 0x7fbbb0070d90 2023-11-08T08:21:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbb9c0682f0 msgr2=0x7fbb9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbba4000c00 tx=0x7fbba4008030 comp rx=0 tx=0).stop 2023-11-08T08:21:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 msgr2=0x7fbbb00c4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 0x7fbbb00c4950 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbba8007e60 tx=0x7fbba800e3c0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 shutdown_connections 2023-11-08T08:21:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb0070d90 0x7fbbb00c4950 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbbb006a270 0x7fbbb00cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 --2- 172.21.15.116:0/737908145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb006da40 0x7fbbb00c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 >> 172.21.15.116:0/737908145 conn(0x7fbbb006c010 msgr2=0x7fbbb00b5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 shutdown_connections 2023-11-08T08:21:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:53.602+0000 7fbbb6371700 1 -- 172.21.15.116:0/737908145 wait complete. 2023-11-08T08:21:53.615 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... FAIL 2023-11-08T08:21:53.615 INFO:tasks.cephfs_test_runner: 2023-11-08T08:21:53.615 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-08T08:21:53.616 INFO:tasks.cephfs_test_runner:FAIL: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:21:53.616 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared 2023-11-08T08:21:53.616 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-08T08:21:53.617 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-08T08:21:53.617 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa/tasks/cephfs/test_forward_scrub.py", line 496, in test_health_status_after_backtrace_repair 2023-11-08T08:21:53.617 INFO:tasks.cephfs_test_runner: self.assertIn("MDS_DAMAGE", self.mds_cluster.mon_manager.get_mon_health()['checks']) 2023-11-08T08:21:53.617 INFO:tasks.cephfs_test_runner:AssertionError: 'MDS_DAMAGE' not found in {} 2023-11-08T08:21:53.618 INFO:tasks.cephfs_test_runner: 2023-11-08T08:21:53.618 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-08T08:21:53.618 INFO:tasks.cephfs_test_runner:Ran 3 tests in 162.185s 2023-11-08T08:21:53.618 INFO:tasks.cephfs_test_runner: 2023-11-08T08:21:53.619 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2023-11-08T08:21:53.619 INFO:tasks.cephfs_test_runner: 2023-11-08T08:21:53.619 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-08T08:21:53.620 INFO:tasks.cephfs_test_runner:FAIL: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:21:53.620 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared 2023-11-08T08:21:53.620 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-08T08:21:53.620 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-08T08:21:53.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa/tasks/cephfs/test_forward_scrub.py", line 496, in test_health_status_after_backtrace_repair 2023-11-08T08:21:53.621 INFO:tasks.cephfs_test_runner: self.assertIn("MDS_DAMAGE", self.mds_cluster.mon_manager.get_mon_health()['checks']) 2023-11-08T08:21:53.621 INFO:tasks.cephfs_test_runner:AssertionError: 'MDS_DAMAGE' not found in {} 2023-11-08T08:21:53.621 INFO:tasks.cephfs_test_runner: 2023-11-08T08:21:53.622 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:21:53.691 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6ffcb431f28d44b3b00d0c6363878d22 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b820560f30d310ab2570761b9a294523225d6998/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-08T08:21:53.693 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-08T08:21:53.703 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-11-08T08:21:53.711 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-11-08T08:21:53.711 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:53.711 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:53.742 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:21:53.743 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-08T08:21:53.743 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:53.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4415 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-08T08:21:53.802 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:53.802 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:53.825 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:53.826 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2023-11-08T08:21:53.826 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-08T08:21:53.886 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251144k,nr_inodes=4062786,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18105 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253768k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-08T08:21:53.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:53.887 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-08T08:21:53.905 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-08T08:21:53.906 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-08T08:21:53.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:53.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-08T08:21:53.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:53.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-08T08:21:54.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:54.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2023-11-08T08:21:54.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:54.159 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2023-11-08T08:21:54.181 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-08T08:21:54.181 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-08T08:21:54.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:54.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-08T08:21:54.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:54.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-08T08:21:54.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-08T08:21:54.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2023-11-08T08:21:54.390 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-08T08:21:54.400 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-08T08:21:54.401 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:21:54.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.522+0000 7fcc7e4df700 1 Processor -- start 2023-11-08T08:21:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.522+0000 7fcc7e4df700 1 -- start start 2023-11-08T08:21:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc7e4df700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc7815db00 0x7fcc7815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc7e4df700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7815a060 con 0x7fcc7806ce20 2023-11-08T08:21:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc7e4df700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc78159a80 con 0x7fcc780c9230 2023-11-08T08:21:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc7e4df700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc780c9ac0 con 0x7fcc7815db00 2023-11-08T08:21:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc77fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcc7806ce20 0x7fcc7806d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58518/0 (socket says 172.21.15.116:58518) 2023-11-08T08:21:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc77fff700 1 -- 172.21.15.116:0/3487333329 learned_addr learned my addr 172.21.15.116:0/3487333329 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:54.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.523+0000 7fcc777fe700 1 --2- 172.21.15.116:0/3487333329 >> v2:172.21.15.129:3300/0 conn(0x7fcc7815db00 0x7fcc7815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665162354 0 0) 0x7fcc7815a060 con 0x7fcc7806ce20 2023-11-08T08:21:54.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc600035b0 con 0x7fcc7806ce20 2023-11-08T08:21:54.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593728441 0 0) 0x7fcc78159a80 con 0x7fcc780c9230 2023-11-08T08:21:54.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc7815a060 con 0x7fcc780c9230 2023-11-08T08:21:54.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc777fe700 1 -- 172.21.15.116:0/3487333329 >> v1:172.21.15.129:6789/0 conn(0x7fcc780c9230 legacy=0x7fcc780ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649354977 0 0) 0x7fcc600035b0 con 0x7fcc7806ce20 2023-11-08T08:21:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc777fe700 1 -- 172.21.15.116:0/3487333329 >> v1:172.21.15.116:6789/0 conn(0x7fcc7806ce20 legacy=0x7fcc7806d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc777fe700 1 -- 172.21.15.116:0/3487333329 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78159cf0 con 0x7fcc7815db00 2023-11-08T08:21:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc777fe700 1 --2- 172.21.15.116:0/3487333329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815db00 0x7fcc7815a4f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009840 tx=0x7fcc68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d115b0c889bf1a6a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c001670 con 0x7fcc7815db00 2023-11-08T08:21:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68004030 con 0x7fcc7815db00 2023-11-08T08:21:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.524+0000 7fcc76ffd700 1 -- 172.21.15.116:0/3487333329 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc680043c0 con 0x7fcc7815db00 2023-11-08T08:21:54.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 -- 172.21.15.116:0/3487333329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815db00 msgr2=0x7fcc7815a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/3487333329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815db00 0x7fcc7815a4f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009840 tx=0x7fcc68009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:54.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 -- 172.21.15.116:0/3487333329 shutdown_connections 2023-11-08T08:21:54.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/3487333329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815db00 0x7fcc7815a4f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 -- 172.21.15.116:0/3487333329 >> 172.21.15.116:0/3487333329 conn(0x7fcc780cb660 msgr2=0x7fcc780cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:54.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 -- 172.21.15.116:0/3487333329 shutdown_connections 2023-11-08T08:21:54.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.525+0000 7fcc7e4df700 1 -- 172.21.15.116:0/3487333329 wait complete. 2023-11-08T08:21:54.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.526+0000 7fcc7e4df700 1 Processor -- start 2023-11-08T08:21:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.526+0000 7fcc7e4df700 1 -- start start 2023-11-08T08:21:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7e4df700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcc7806ce20 0x7fcc781f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7e4df700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc780c9230 0x7fcc780c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7e4df700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 0x7fcc7815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7e4df700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780c6620 con 0x7fcc7815db00 2023-11-08T08:21:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7e4df700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780c6900 con 0x7fcc780c9230 2023-11-08T08:21:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7e4df700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780c6be0 con 0x7fcc7806ce20 2023-11-08T08:21:54.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 0x7fcc7815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 0x7fcc7815c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53812/0 (socket says 172.21.15.116:53812) 2023-11-08T08:21:54.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 -- 172.21.15.116:0/749180316 learned_addr learned my addr 172.21.15.116:0/749180316 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc77fff700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcc7806ce20 0x7fcc781f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc777fe700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc780c9230 0x7fcc780c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 -- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcc7806ce20 msgr2=0x7fcc781f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcc7806ce20 0x7fcc781f36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 -- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc780c9230 msgr2=0x7fcc780c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc780c9230 0x7fcc780c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 -- 172.21.15.116:0/749180316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680094a0 con 0x7fcc7815db00 2023-11-08T08:21:54.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc777fe700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc780c9230 0x7fcc780c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:54.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc77fff700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcc7806ce20 0x7fcc781f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:54.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.527+0000 7fcc7ca7c700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 0x7fcc7815c700 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000cc20 tx=0x7fcc7000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:54.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.528+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000ac20 con 0x7fcc7815db00 2023-11-08T08:21:54.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.528+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7000ad80 con 0x7fcc7815db00 2023-11-08T08:21:54.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.528+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc780c9da0 con 0x7fcc7815db00 2023-11-08T08:21:54.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.528+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70003c20 con 0x7fcc7815db00 2023-11-08T08:21:54.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.528+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc78159790 con 0x7fcc7815db00 2023-11-08T08:21:54.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.529+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc780c9f30 con 0x7fcc7815db00 2023-11-08T08:21:54.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.529+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc70010050 con 0x7fcc7815db00 2023-11-08T08:21:54.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.529+0000 7fcc74ff9700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcc640682f0 0x7fcc6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.529+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fcc70085a00 con 0x7fcc7815db00 2023-11-08T08:21:54.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.530+0000 7fcc77fff700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcc640682f0 0x7fcc6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.530+0000 7fcc77fff700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcc640682f0 0x7fcc6406a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007960 tx=0x7fcc6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:54.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.534+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc70052560 con 0x7fcc7815db00 2023-11-08T08:21:54.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.711+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc780743f0 con 0x7fcc640682f0 2023-11-08T08:21:54.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.713+0000 7fcc74ff9700 1 -- 172.21.15.116:0/749180316 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72961 (secure 0 0 0) 0x7fcc780743f0 con 0x7fcc640682f0 2023-11-08T08:21:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:54.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcc640682f0 msgr2=0x7fcc6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcc640682f0 0x7fcc6406a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007960 tx=0x7fcc6c009040 comp rx=0 tx=0).stop 2023-11-08T08:21:54.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 msgr2=0x7fcc7815c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 0x7fcc7815c700 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000cc20 tx=0x7fcc7000a060 comp rx=0 tx=0).stop 2023-11-08T08:21:54.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 shutdown_connections 2023-11-08T08:21:54.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7815db00 0x7fcc7815c700 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcc7806ce20 0x7fcc781f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fcc640682f0 0x7fcc6406a7a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 --2- 172.21.15.116:0/749180316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc780c9230 0x7fcc780c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 >> 172.21.15.116:0/749180316 conn(0x7fcc780cb660 msgr2=0x7fcc780b3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:54.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 shutdown_connections 2023-11-08T08:21:54.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.716+0000 7fcc7e4df700 1 -- 172.21.15.116:0/749180316 wait complete. 2023-11-08T08:21:54.721 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:21:54.729 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2023-11-08T08:21:54.395168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":78416,"kb_used_data":6544,"kb_used_omap":0,"kb_used_meta":71808,"kb_avail":754896304,"statfs":{"total":773094113280,"available":773013815296,"internally_reserved":0,"allocated":6701056,"data_stored":3577735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73531392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12289,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-6,"num_read_kb":-5,"num_write":-71,"num_write_kb":-36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001309"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291175+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:21:33.291175+0000","last_peered":"2023-11-08T08:21:33.291175+0000","last_clean":"2023-11-08T08:21:33.291175+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:33.291175+0000","last_undegraded":"2023-11-08T08:21:33.291175+0000","last_fullsized":"2023-11-08T08:21:33.291175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291246+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:21:33.291246+0000","last_peered":"2023-11-08T08:21:33.291246+0000","last_clean":"2023-11-08T08:21:33.291246+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:21:33.291246+0000","last_undegraded":"2023-11-08T08:21:33.291246+0000","last_fullsized":"2023-11-08T08:21:33.291246+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176216+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:21:33.176216+0000","last_peered":"2023-11-08T08:21:33.176216+0000","last_clean":"2023-11-08T08:21:33.176216+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:21:33.176216+0000","last_undegraded":"2023-11-08T08:21:33.176216+0000","last_fullsized":"2023-11-08T08:21:33.176216+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290355+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:21:33.290355+0000","last_peered":"2023-11-08T08:21:33.290355+0000","last_clean":"2023-11-08T08:21:33.290355+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:21:33.290355+0000","last_undegraded":"2023-11-08T08:21:33.290355+0000","last_fullsized":"2023-11-08T08:21:33.290355+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096417+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:21:33.096417+0000","last_peered":"2023-11-08T08:21:33.096417+0000","last_clean":"2023-11-08T08:21:33.096417+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:21:33.096417+0000","last_undegraded":"2023-11-08T08:21:33.096417+0000","last_fullsized":"2023-11-08T08:21:33.096417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098302+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:21:33.098302+0000","last_peered":"2023-11-08T08:21:33.098302+0000","last_clean":"2023-11-08T08:21:33.098302+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:33.098302+0000","last_undegraded":"2023-11-08T08:21:33.098302+0000","last_fullsized":"2023-11-08T08:21:33.098302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290472+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:21:33.290472+0000","last_peered":"2023-11-08T08:21:33.290472+0000","last_clean":"2023-11-08T08:21:33.290472+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:21:33.290472+0000","last_undegraded":"2023-11-08T08:21:33.290472+0000","last_fullsized":"2023-11-08T08:21:33.290472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290593+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:21:33.290593+0000","last_peered":"2023-11-08T08:21:33.290593+0000","last_clean":"2023-11-08T08:21:33.290593+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:21:33.290593+0000","last_undegraded":"2023-11-08T08:21:33.290593+0000","last_fullsized":"2023-11-08T08:21:33.290593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098536+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:21:33.098536+0000","last_peered":"2023-11-08T08:21:33.098536+0000","last_clean":"2023-11-08T08:21:33.098536+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:21:33.098536+0000","last_undegraded":"2023-11-08T08:21:33.098536+0000","last_fullsized":"2023-11-08T08:21:33.098536+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9316,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362524,"statfs":{"total":96636764160,"available":96627224576,"internally_reserved":0,"allocated":684032,"data_stored":299123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":1.611,"15min":1.611},"last":0.46200000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.67100000000000004,"15min":0.72499999999999998},"min":{"1min":0.35699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.73899999999999999,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.68200000000000005,"15min":0.73099999999999998},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81299999999999994,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.56100000000000005}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73499999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55900000000000005,"15min":0.60299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59699999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.316},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57099999999999995,"15min":0.60099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42799999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61499999999999999,"5min":0.872,"15min":0.872},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52500000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70699999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54900000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9520,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362320,"statfs":{"total":96636764160,"available":96627015680,"internally_reserved":0,"allocated":696320,"data_stored":303571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91200000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.115,"15min":1.115},"last":0.42299999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.44400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":1.089,"15min":1.089},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.996,"15min":0.996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.135},"last":0.496},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.66000000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.67300000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58199999999999996}]},{"osd":7,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.747,"15min":0.747},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9448,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362392,"statfs":{"total":96636764160,"available":96627089408,"internally_reserved":0,"allocated":688128,"data_stored":299157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.311,"15min":0.308},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.379,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.309},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.216}]},{"osd":2,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":1.149,"15min":1.149},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.443,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.27200000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48399999999999999,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61099999999999999,"15min":0.56499999999999995},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70799999999999996,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.66200000000000003,"15min":0.63200000000000001},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.67700000000000005,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.52900000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77600000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.53900000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67300000000000004,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.58699999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67600000000000005,"15min":0.629},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83199999999999996,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69299999999999995,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9380,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362460,"statfs":{"total":96636764160,"available":96627159040,"internally_reserved":0,"allocated":684032,"data_stored":299123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.45100000000000001,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45500000000000002,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51800000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.311}]},{"osd":2,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":2.032,"15min":2.032},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41099999999999998,"15min":0.42899999999999999},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60099999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.30399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":1.788,"15min":1.788},"last":0.373},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79600000000000004,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":1.002,"15min":1.002},"last":0.61499999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85199999999999998,"5min":1.825,"15min":1.825},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72699999999999998,"5min":1.75,"15min":1.75},"last":0.59399999999999997}]},{"osd":6,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.56399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9644,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362196,"statfs":{"total":96636764160,"available":96626888704,"internally_reserved":0,"allocated":692224,"data_stored":299179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.19,"15min":0.19},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.311},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.314,"15min":0.311},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.311,"15min":0.309},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.40699999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.377}]},{"osd":3,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.35199999999999998,"15min":0.34200000000000003},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.48499999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.34899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.35399999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81000000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.443}]},{"osd":6,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":1.294,"15min":1.294},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79000000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.56200000000000006},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.46200000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81499999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10596,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361244,"statfs":{"total":96636764160,"available":96625913856,"internally_reserved":0,"allocated":1273856,"data_stored":883217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34399999999999997,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":1.319,"15min":1.319},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38900000000000001,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.46999999999999997,"5min":1.915,"15min":1.915},"last":0.435}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.39900000000000002,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.45200000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.50700000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55500000000000005,"5min":1.278,"15min":1.278},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76200000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79400000000000004,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72399999999999998,"5min":1.371,"15min":1.371},"last":0.70699999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76400000000000001,"5min":1.857,"15min":1.857},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78800000000000003,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.71799999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.66000000000000003,"15min":0.67300000000000004},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80900000000000005,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.53100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9820,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94362020,"statfs":{"total":96636764160,"available":96626708480,"internally_reserved":0,"allocated":675840,"data_stored":293255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.91800000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.41099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.008,"5min":1.073,"15min":1.073},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.47799999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71999999999999997,"5min":1.492,"15min":1.492},"last":0.54100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51900000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.378},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39700000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67100000000000004,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.434}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10692,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361148,"statfs":{"total":96636764160,"available":96625815552,"internally_reserved":0,"allocated":1306624,"data_stored":901110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.85999999999999999,"15min":0.97199999999999998},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8069999999999999,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.629},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82699999999999996,"15min":0.95099999999999996},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.90200000000000002},"min":{"1min":0.48599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.718,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.89400000000000002},"min":{"1min":0.42999999999999999,"5min":0.378,"15min":0.378},"max":{"1min":2.8559999999999999,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.90700000000000003},"min":{"1min":0.47899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.1139999999999999,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":4.891,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.89300000000000002,"15min":1.0089999999999999},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.0579999999999998,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75700000000000001,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.54900000000000004,"15min":0.51300000000000001},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.524,"5min":5.524,"15min":5.524},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":0.66500000000000004}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:21:54.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:21:54.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.850+0000 7f393ecaf700 1 Processor -- start 2023-11-08T08:21:54.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.850+0000 7f393ecaf700 1 -- start start 2023-11-08T08:21:54.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393ecaf700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f393806a270 0x7f393806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393ecaf700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f393806e570 0x7f3938070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393ecaf700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393806ab80 con 0x7f393815cc80 2023-11-08T08:21:54.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393ecaf700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39380b2490 con 0x7f393806e570 2023-11-08T08:21:54.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393ecaf700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393806e990 con 0x7f393806a270 2023-11-08T08:21:54.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f393806e570 0x7f3938070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f393806e570 0x7f3938070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53828/0 (socket says 172.21.15.116:53828) 2023-11-08T08:21:54.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 -- 172.21.15.116:0/686523707 learned_addr learned my addr 172.21.15.116:0/686523707 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:54.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393ca4b700 1 --2- 172.21.15.116:0/686523707 >> v2:172.21.15.129:3300/0 conn(0x7f393806a270 0x7f393806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 -- 172.21.15.116:0/686523707 >> v2:172.21.15.129:3300/0 conn(0x7f393806a270 msgr2=0x7f393806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:54.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 --2- 172.21.15.116:0/686523707 >> v2:172.21.15.129:3300/0 conn(0x7f393806a270 0x7f393806a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 -- 172.21.15.116:0/686523707 >> v1:172.21.15.129:6789/0 conn(0x7f393815cc80 legacy=0x7f393806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 -- 172.21.15.116:0/686523707 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39380b1810 con 0x7f393806e570 2023-11-08T08:21:54.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f393d24c700 1 --2- 172.21.15.116:0/686523707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806e570 0x7f3938070d20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3938067380 tx=0x7f393000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a685e44ea68db46 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:54.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f39377fe700 1 -- 172.21.15.116:0/686523707 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393000e070 con 0x7f393806e570 2023-11-08T08:21:54.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f39377fe700 1 -- 172.21.15.116:0/686523707 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393000b8d0 con 0x7f393806e570 2023-11-08T08:21:54.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.851+0000 7f39377fe700 1 -- 172.21.15.116:0/686523707 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393000bc80 con 0x7f393806e570 2023-11-08T08:21:54.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 -- 172.21.15.116:0/686523707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806e570 msgr2=0x7f3938070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 --2- 172.21.15.116:0/686523707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806e570 0x7f3938070d20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3938067380 tx=0x7f393000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:54.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 -- 172.21.15.116:0/686523707 shutdown_connections 2023-11-08T08:21:54.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 --2- 172.21.15.116:0/686523707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806e570 0x7f3938070d20 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 --2- 172.21.15.116:0/686523707 >> v2:172.21.15.129:3300/0 conn(0x7f393806a270 0x7f393806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 -- 172.21.15.116:0/686523707 >> 172.21.15.116:0/686523707 conn(0x7f393806c010 msgr2=0x7f39380cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:54.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 -- 172.21.15.116:0/686523707 shutdown_connections 2023-11-08T08:21:54.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.852+0000 7f393ecaf700 1 -- 172.21.15.116:0/686523707 wait complete. 2023-11-08T08:21:54.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.853+0000 7f393ecaf700 1 Processor -- start 2023-11-08T08:21:54.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.853+0000 7f393ecaf700 1 -- start start 2023-11-08T08:21:54.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ecaf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 0x7f3938152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ecaf700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393806e570 0x7f393806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ecaf700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393815cc80 0x7f393806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ecaf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393815e7c0 con 0x7f393806a270 2023-11-08T08:21:54.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ecaf700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393815ebe0 con 0x7f393815cc80 2023-11-08T08:21:54.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ecaf700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393815db40 con 0x7f393806e570 2023-11-08T08:21:54.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ca4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 0x7f3938152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ca4b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 0x7f3938152a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53846/0 (socket says 172.21.15.116:53846) 2023-11-08T08:21:54.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393ca4b700 1 -- 172.21.15.116:0/2242481839 learned_addr learned my addr 172.21.15.116:0/2242481839 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:54.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f3937fff700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393806e570 0x7f393806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.854+0000 7f393d24c700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393815cc80 0x7f393806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ca4b700 1 -- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393806e570 msgr2=0x7f393806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ca4b700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393806e570 0x7f393806ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ca4b700 1 -- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393815cc80 msgr2=0x7f393806d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:54.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ca4b700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393815cc80 0x7f393806d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:54.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ca4b700 1 -- 172.21.15.116:0/2242481839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3930009b30 con 0x7f393806a270 2023-11-08T08:21:54.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393d24c700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393815cc80 0x7f393806d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:54.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ca4b700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 0x7f3938152a90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f392c00e8a0 tx=0x7f392c00ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:54.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c009590 con 0x7f393806a270 2023-11-08T08:21:54.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c013680 con 0x7f393806a270 2023-11-08T08:21:54.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c0139f0 con 0x7f393806a270 2023-11-08T08:21:54.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39380b1dd0 con 0x7f393806a270 2023-11-08T08:21:54.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.855+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39380b1af0 con 0x7f393806a270 2023-11-08T08:21:54.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.856+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39380b20b0 con 0x7f393806a270 2023-11-08T08:21:54.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.856+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f392c019030 con 0x7f393806a270 2023-11-08T08:21:54.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.856+0000 7f39357fa700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f391c0682f0 0x7f391c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:54.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.857+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f392c085730 con 0x7f393806a270 2023-11-08T08:21:54.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.857+0000 7f3937fff700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f391c0682f0 0x7f391c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:54.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.857+0000 7f3937fff700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f391c0682f0 0x7f391c06a7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f39380c3670 tx=0x7f3924009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:54.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:54.861+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f392c052290 con 0x7f393806a270 2023-11-08T08:21:55.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.038+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f393815ea30 con 0x7f391c0682f0 2023-11-08T08:21:55.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.041+0000 7f39357fa700 1 -- 172.21.15.116:0/2242481839 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72961 (secure 0 0 0) 0x7f393815ea30 con 0x7f391c0682f0 2023-11-08T08:21:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:55.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.042+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f391c0682f0 msgr2=0x7f391c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f391c0682f0 0x7f391c06a7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f39380c3670 tx=0x7f3924009040 comp rx=0 tx=0).stop 2023-11-08T08:21:55.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 msgr2=0x7f3938152a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 0x7f3938152a90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f392c00e8a0 tx=0x7f392c00ebb0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 shutdown_connections 2023-11-08T08:21:55.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f391c0682f0 0x7f391c06a7a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f393815cc80 0x7f393806d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f393806a270 0x7f3938152a90 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 --2- 172.21.15.116:0/2242481839 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f393806e570 0x7f393806ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 >> 172.21.15.116:0/2242481839 conn(0x7f393806c010 msgr2=0x7f39380cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:55.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 shutdown_connections 2023-11-08T08:21:55.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.043+0000 7f393ecaf700 1 -- 172.21.15.116:0/2242481839 wait complete. 2023-11-08T08:21:55.048 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:21:55.055 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2023-11-08T08:21:54.395168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":78416,"kb_used_data":6544,"kb_used_omap":0,"kb_used_meta":71808,"kb_avail":754896304,"statfs":{"total":773094113280,"available":773013815296,"internally_reserved":0,"allocated":6701056,"data_stored":3577735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73531392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12289,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-6,"num_read_kb":-5,"num_write":-71,"num_write_kb":-36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001309"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291175+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:21:33.291175+0000","last_peered":"2023-11-08T08:21:33.291175+0000","last_clean":"2023-11-08T08:21:33.291175+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:33.291175+0000","last_undegraded":"2023-11-08T08:21:33.291175+0000","last_fullsized":"2023-11-08T08:21:33.291175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291246+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:21:33.291246+0000","last_peered":"2023-11-08T08:21:33.291246+0000","last_clean":"2023-11-08T08:21:33.291246+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:21:33.291246+0000","last_undegraded":"2023-11-08T08:21:33.291246+0000","last_fullsized":"2023-11-08T08:21:33.291246+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176216+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:21:33.176216+0000","last_peered":"2023-11-08T08:21:33.176216+0000","last_clean":"2023-11-08T08:21:33.176216+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:21:33.176216+0000","last_undegraded":"2023-11-08T08:21:33.176216+0000","last_fullsized":"2023-11-08T08:21:33.176216+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290355+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:21:33.290355+0000","last_peered":"2023-11-08T08:21:33.290355+0000","last_clean":"2023-11-08T08:21:33.290355+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:21:33.290355+0000","last_undegraded":"2023-11-08T08:21:33.290355+0000","last_fullsized":"2023-11-08T08:21:33.290355+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096417+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:21:33.096417+0000","last_peered":"2023-11-08T08:21:33.096417+0000","last_clean":"2023-11-08T08:21:33.096417+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:21:33.096417+0000","last_undegraded":"2023-11-08T08:21:33.096417+0000","last_fullsized":"2023-11-08T08:21:33.096417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098302+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:21:33.098302+0000","last_peered":"2023-11-08T08:21:33.098302+0000","last_clean":"2023-11-08T08:21:33.098302+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:33.098302+0000","last_undegraded":"2023-11-08T08:21:33.098302+0000","last_fullsized":"2023-11-08T08:21:33.098302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290472+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:21:33.290472+0000","last_peered":"2023-11-08T08:21:33.290472+0000","last_clean":"2023-11-08T08:21:33.290472+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:21:33.290472+0000","last_undegraded":"2023-11-08T08:21:33.290472+0000","last_fullsized":"2023-11-08T08:21:33.290472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290593+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:21:33.290593+0000","last_peered":"2023-11-08T08:21:33.290593+0000","last_clean":"2023-11-08T08:21:33.290593+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:21:33.290593+0000","last_undegraded":"2023-11-08T08:21:33.290593+0000","last_fullsized":"2023-11-08T08:21:33.290593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098536+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:21:33.098536+0000","last_peered":"2023-11-08T08:21:33.098536+0000","last_clean":"2023-11-08T08:21:33.098536+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:21:33.098536+0000","last_undegraded":"2023-11-08T08:21:33.098536+0000","last_fullsized":"2023-11-08T08:21:33.098536+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9316,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362524,"statfs":{"total":96636764160,"available":96627224576,"internally_reserved":0,"allocated":684032,"data_stored":299123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":1.611,"15min":1.611},"last":0.46200000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.67100000000000004,"15min":0.72499999999999998},"min":{"1min":0.35699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.73899999999999999,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.68200000000000005,"15min":0.73099999999999998},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81299999999999994,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.56100000000000005}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73499999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55900000000000005,"15min":0.60299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59699999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.316},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57099999999999995,"15min":0.60099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42799999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61499999999999999,"5min":0.872,"15min":0.872},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52500000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70699999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54900000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9520,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362320,"statfs":{"total":96636764160,"available":96627015680,"internally_reserved":0,"allocated":696320,"data_stored":303571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91200000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.115,"15min":1.115},"last":0.42299999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.44400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":1.089,"15min":1.089},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.996,"15min":0.996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.135},"last":0.496},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.66000000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.67300000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58199999999999996}]},{"osd":7,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.747,"15min":0.747},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9448,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362392,"statfs":{"total":96636764160,"available":96627089408,"internally_reserved":0,"allocated":688128,"data_stored":299157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.311,"15min":0.308},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.379,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.309},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.216}]},{"osd":2,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":1.149,"15min":1.149},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.443,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.27200000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48399999999999999,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61099999999999999,"15min":0.56499999999999995},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70799999999999996,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.66200000000000003,"15min":0.63200000000000001},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.67700000000000005,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.52900000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77600000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.53900000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67300000000000004,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.58699999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67600000000000005,"15min":0.629},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83199999999999996,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69299999999999995,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9380,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362460,"statfs":{"total":96636764160,"available":96627159040,"internally_reserved":0,"allocated":684032,"data_stored":299123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.45100000000000001,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45500000000000002,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51800000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.311}]},{"osd":2,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":2.032,"15min":2.032},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41099999999999998,"15min":0.42899999999999999},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60099999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.30399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":1.788,"15min":1.788},"last":0.373},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79600000000000004,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":1.002,"15min":1.002},"last":0.61499999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85199999999999998,"5min":1.825,"15min":1.825},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72699999999999998,"5min":1.75,"15min":1.75},"last":0.59399999999999997}]},{"osd":6,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.56399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9644,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362196,"statfs":{"total":96636764160,"available":96626888704,"internally_reserved":0,"allocated":692224,"data_stored":299179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.19,"15min":0.19},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.311},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.314,"15min":0.311},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.311,"15min":0.309},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.40699999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.377}]},{"osd":3,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.35199999999999998,"15min":0.34200000000000003},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.48499999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.34899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.35399999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81000000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.443}]},{"osd":6,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":1.294,"15min":1.294},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79000000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.56200000000000006},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.46200000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81499999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10596,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361244,"statfs":{"total":96636764160,"available":96625913856,"internally_reserved":0,"allocated":1273856,"data_stored":883217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34399999999999997,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":1.319,"15min":1.319},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38900000000000001,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.46999999999999997,"5min":1.915,"15min":1.915},"last":0.435}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.39900000000000002,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.45200000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.50700000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55500000000000005,"5min":1.278,"15min":1.278},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76200000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79400000000000004,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72399999999999998,"5min":1.371,"15min":1.371},"last":0.70699999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76400000000000001,"5min":1.857,"15min":1.857},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78800000000000003,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.71799999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.66000000000000003,"15min":0.67300000000000004},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80900000000000005,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.53100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9820,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94362020,"statfs":{"total":96636764160,"available":96626708480,"internally_reserved":0,"allocated":675840,"data_stored":293255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.91800000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.41099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.008,"5min":1.073,"15min":1.073},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.47799999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71999999999999997,"5min":1.492,"15min":1.492},"last":0.54100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51900000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.378},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39700000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67100000000000004,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.434}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10692,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361148,"statfs":{"total":96636764160,"available":96625815552,"internally_reserved":0,"allocated":1306624,"data_stored":901110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.85999999999999999,"15min":0.97199999999999998},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8069999999999999,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.629},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82699999999999996,"15min":0.95099999999999996},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.90200000000000002},"min":{"1min":0.48599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.718,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.89400000000000002},"min":{"1min":0.42999999999999999,"5min":0.378,"15min":0.378},"max":{"1min":2.8559999999999999,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.90700000000000003},"min":{"1min":0.47899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.1139999999999999,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":4.891,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.89300000000000002,"15min":1.0089999999999999},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.0579999999999998,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75700000000000001,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.54900000000000004,"15min":0.51300000000000001},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.524,"5min":5.524,"15min":5.524},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":0.66500000000000004}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:21:55.058 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-08T08:21:55.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:21:55.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.177+0000 7fb4de456700 1 Processor -- start 2023-11-08T08:21:55.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.177+0000 7fb4de456700 1 -- start start 2023-11-08T08:21:55.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4de456700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb4d806ab10 0x7fb4d806aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4de456700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 0x7fb4d8160a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4de456700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d80d1ee0 con 0x7fb4d80cd5a0 2023-11-08T08:21:55.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4de456700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d806b420 con 0x7fb4d806ab10 2023-11-08T08:21:55.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4de456700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80cdb10 con 0x7fb4d80d0450 2023-11-08T08:21:55.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4d77fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb4d80cd5a0 0x7fb4d8160300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58530/0 (socket says 172.21.15.116:58530) 2023-11-08T08:21:55.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4d77fe700 1 -- 172.21.15.116:0/2242727160 learned_addr learned my addr 172.21.15.116:0/2242727160 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:55.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331737698 0 0) 0x7fb4d80d1ee0 con 0x7fb4d80cd5a0 2023-11-08T08:21:55.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c00035b0 con 0x7fb4d80cd5a0 2023-11-08T08:21:55.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 0x7fb4d8160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.178+0000 7fb4d7fff700 1 --2- 172.21.15.116:0/2242727160 >> v2:172.21.15.129:3300/0 conn(0x7fb4d806ab10 0x7fb4d806aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3911340050 0 0) 0x7fb4c00035b0 con 0x7fb4d80cd5a0 2023-11-08T08:21:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 msgr2=0x7fb4d8160a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 0x7fb4d8160a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 >> v2:172.21.15.129:3300/0 conn(0x7fb4d806ab10 msgr2=0x7fb4d806aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 --2- 172.21.15.116:0/2242727160 >> v2:172.21.15.129:3300/0 conn(0x7fb4d806ab10 0x7fb4d806aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d80d1890 con 0x7fb4d80cd5a0 2023-11-08T08:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fb4cc001670 con 0x7fb4d80cd5a0 2023-11-08T08:21:55.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 >> v1:172.21.15.116:6789/0 conn(0x7fb4d80cd5a0 legacy=0x7fb4d8160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0003a00 0x7fb4c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 0x7fb4c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0004d90 0x7fb4c0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d806b420 con 0x7fb4c0004d90 2023-11-08T08:21:55.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80cdb10 con 0x7fb4c0003a00 2023-11-08T08:21:55.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c000ba10 con 0x7fb4d80d0450 2023-11-08T08:21:55.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0004d90 0x7fb4c0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d7fff700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0003a00 0x7fb4c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d77fe700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 0x7fb4c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 -- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 msgr2=0x7fb4c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:55.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80d0450 0x7fb4c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 -- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0003a00 msgr2=0x7fb4c0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:55.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0003a00 0x7fb4c0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 -- 172.21.15.116:0/2242727160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00035b0 con 0x7fb4c0004d90 2023-11-08T08:21:55.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0004d90 0x7fb4c0005200 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009fc0 tx=0x7fb4c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2476511f95b73a4e server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:55.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d000ec30 con 0x7fb4c0004d90 2023-11-08T08:21:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d000f040 con 0x7fb4c0004d90 2023-11-08T08:21:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.179+0000 7fb4d6ffd700 1 -- 172.21.15.116:0/2242727160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0014480 con 0x7fb4c0004d90 2023-11-08T08:21:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4de456700 1 -- 172.21.15.116:0/2242727160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0004d90 msgr2=0x7fb4c0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4de456700 1 --2- 172.21.15.116:0/2242727160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0004d90 0x7fb4c0005200 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009fc0 tx=0x7fb4c0005880 comp rx=0 tx=0).stop 2023-11-08T08:21:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4dc9f3700 1 -- 172.21.15.116:0/2242727160 reap_dead start 2023-11-08T08:21:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4de456700 1 -- 172.21.15.116:0/2242727160 shutdown_connections 2023-11-08T08:21:55.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4de456700 1 -- 172.21.15.116:0/2242727160 >> 172.21.15.116:0/2242727160 conn(0x7fb4d806c520 msgr2=0x7fb4d806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:55.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4de456700 1 -- 172.21.15.116:0/2242727160 shutdown_connections 2023-11-08T08:21:55.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.180+0000 7fb4de456700 1 -- 172.21.15.116:0/2242727160 wait complete. 2023-11-08T08:21:55.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.181+0000 7fb4de456700 1 Processor -- start 2023-11-08T08:21:55.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.181+0000 7fb4de456700 1 -- start start 2023-11-08T08:21:55.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4de456700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0004d90 0x7fb4d80cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4de456700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 0x7fb4d80b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4de456700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80c8400 0x7fb4d80b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4de456700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81f3720 con 0x7fb4d80c7f90 2023-11-08T08:21:55.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4de456700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81f3440 con 0x7fb4c0004d90 2023-11-08T08:21:55.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4de456700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80cb1a0 con 0x7fb4d80c8400 2023-11-08T08:21:55.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4d77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 0x7fb4d80b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4d77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 0x7fb4d80b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53852/0 (socket says 172.21.15.116:53852) 2023-11-08T08:21:55.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4d77fe700 1 -- 172.21.15.116:0/2133035393 learned_addr learned my addr 172.21.15.116:0/2133035393 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:55.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4d7fff700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0004d90 0x7fb4d80cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80c8400 0x7fb4d80b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4d77fe700 1 -- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80c8400 msgr2=0x7fb4d80b64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.182+0000 7fb4d77fe700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80c8400 0x7fb4d80b64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d77fe700 1 -- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0004d90 msgr2=0x7fb4d80cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d77fe700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0004d90 0x7fb4d80cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d77fe700 1 -- 172.21.15.116:0/2133035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d815cad0 con 0x7fb4d80c7f90 2023-11-08T08:21:55.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4dc9f3700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80c8400 0x7fb4d80b64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:55.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d7fff700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0004d90 0x7fb4d80cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:55.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d77fe700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 0x7fb4d80b5f90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc000fb0 tx=0x7fb4cc0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:55.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc00f840 con 0x7fb4d80c7f90 2023-11-08T08:21:55.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc00f9c0 con 0x7fb4d80c7f90 2023-11-08T08:21:55.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d80d1b70 con 0x7fb4d80c7f90 2023-11-08T08:21:55.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc00fd70 con 0x7fb4d80c7f90 2023-11-08T08:21:55.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.183+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d80b1ec0 con 0x7fb4d80c7f90 2023-11-08T08:21:55.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.184+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d80743f0 con 0x7fb4d80c7f90 2023-11-08T08:21:55.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.184+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb4cc01d460 con 0x7fb4d80c7f90 2023-11-08T08:21:55.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.184+0000 7fb4d4ff9700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4c40682f0 0x7fb4c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.185+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fb4cc085be0 con 0x7fb4d80c7f90 2023-11-08T08:21:55.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.185+0000 7fb4d7fff700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4c40682f0 0x7fb4c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.185+0000 7fb4d7fff700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4c40682f0 0x7fb4c406a7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009c60 tx=0x7fb4c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:55.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.189+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb4cc0526e0 con 0x7fb4d80c7f90 2023-11-08T08:21:55.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.362+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4d80d2c80 con 0x7fb4c40682f0 2023-11-08T08:21:55.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.365+0000 7fb4d4ff9700 1 -- 172.21.15.116:0/2133035393 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72961 (secure 0 0 0) 0x7fb4d80d2c80 con 0x7fb4c40682f0 2023-11-08T08:21:55.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:55.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4c40682f0 msgr2=0x7fb4c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4c40682f0 0x7fb4c406a7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009c60 tx=0x7fb4c8008040 comp rx=0 tx=0).stop 2023-11-08T08:21:55.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 msgr2=0x7fb4d80b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 0x7fb4d80b5f90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc000fb0 tx=0x7fb4cc0076a0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 shutdown_connections 2023-11-08T08:21:55.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb4d80c8400 0x7fb4d80b64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb4c40682f0 0x7fb4c406a7a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4c0004d90 0x7fb4d80cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 --2- 172.21.15.116:0/2133035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d80c7f90 0x7fb4d80b5f90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 >> 172.21.15.116:0/2133035393 conn(0x7fb4d806c520 msgr2=0x7fb4d80b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:55.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 shutdown_connections 2023-11-08T08:21:55.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.367+0000 7fb4de456700 1 -- 172.21.15.116:0/2133035393 wait complete. 2023-11-08T08:21:55.372 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:21:55.380 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2023-11-08T08:21:54.395168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":78416,"kb_used_data":6544,"kb_used_omap":0,"kb_used_meta":71808,"kb_avail":754896304,"statfs":{"total":773094113280,"available":773013815296,"internally_reserved":0,"allocated":6701056,"data_stored":3577735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73531392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12289,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-6,"num_read_kb":-5,"num_write":-71,"num_write_kb":-36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001309"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291175+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:21:33.291175+0000","last_peered":"2023-11-08T08:21:33.291175+0000","last_clean":"2023-11-08T08:21:33.291175+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:33.291175+0000","last_undegraded":"2023-11-08T08:21:33.291175+0000","last_fullsized":"2023-11-08T08:21:33.291175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.291246+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:21:33.291246+0000","last_peered":"2023-11-08T08:21:33.291246+0000","last_clean":"2023-11-08T08:21:33.291246+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:21:33.291246+0000","last_undegraded":"2023-11-08T08:21:33.291246+0000","last_fullsized":"2023-11-08T08:21:33.291246+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.176216+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:21:33.176216+0000","last_peered":"2023-11-08T08:21:33.176216+0000","last_clean":"2023-11-08T08:21:33.176216+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:21:33.176216+0000","last_undegraded":"2023-11-08T08:21:33.176216+0000","last_fullsized":"2023-11-08T08:21:33.176216+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":199,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290355+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:21:33.290355+0000","last_peered":"2023-11-08T08:21:33.290355+0000","last_clean":"2023-11-08T08:21:33.290355+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:21:33.290355+0000","last_undegraded":"2023-11-08T08:21:33.290355+0000","last_fullsized":"2023-11-08T08:21:33.290355+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.096417+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:21:33.096417+0000","last_peered":"2023-11-08T08:21:33.096417+0000","last_clean":"2023-11-08T08:21:33.096417+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:21:33.096417+0000","last_undegraded":"2023-11-08T08:21:33.096417+0000","last_fullsized":"2023-11-08T08:21:33.096417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098302+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:21:33.098302+0000","last_peered":"2023-11-08T08:21:33.098302+0000","last_clean":"2023-11-08T08:21:33.098302+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:33.098302+0000","last_undegraded":"2023-11-08T08:21:33.098302+0000","last_fullsized":"2023-11-08T08:21:33.098302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290472+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:21:33.290472+0000","last_peered":"2023-11-08T08:21:33.290472+0000","last_clean":"2023-11-08T08:21:33.290472+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:21:33.290472+0000","last_undegraded":"2023-11-08T08:21:33.290472+0000","last_fullsized":"2023-11-08T08:21:33.290472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.290593+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:21:33.290593+0000","last_peered":"2023-11-08T08:21:33.290593+0000","last_clean":"2023-11-08T08:21:33.290593+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:21:33.290593+0000","last_undegraded":"2023-11-08T08:21:33.290593+0000","last_fullsized":"2023-11-08T08:21:33.290593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":44,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-11-08T08:21:33.098536+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:21:33.098536+0000","last_peered":"2023-11-08T08:21:33.098536+0000","last_clean":"2023-11-08T08:21:33.098536+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:21:33.098536+0000","last_undegraded":"2023-11-08T08:21:33.098536+0000","last_fullsized":"2023-11-08T08:21:33.098536+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9316,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":8640,"kb_avail":94362524,"statfs":{"total":96636764160,"available":96627224576,"internally_reserved":0,"allocated":684032,"data_stored":299123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8847360},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":1.611,"15min":1.611},"last":0.46200000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.67100000000000004,"15min":0.72499999999999998},"min":{"1min":0.35699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.73899999999999999,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.68200000000000005,"15min":0.73099999999999998},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81299999999999994,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.56100000000000005}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73499999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55900000000000005,"15min":0.60299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59699999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.316},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57099999999999995,"15min":0.60099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42799999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61499999999999999,"5min":0.872,"15min":0.872},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52500000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70699999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54900000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9520,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362320,"statfs":{"total":96636764160,"available":96627015680,"internally_reserved":0,"allocated":696320,"data_stored":303571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91200000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.115,"15min":1.115},"last":0.42299999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.44400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":1.089,"15min":1.089},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.996,"15min":0.996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.135},"last":0.496},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.66000000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.67300000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58199999999999996}]},{"osd":7,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.747,"15min":0.747},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9448,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362392,"statfs":{"total":96636764160,"available":96627089408,"internally_reserved":0,"allocated":688128,"data_stored":299157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.311,"15min":0.308},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.379,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.309},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.216}]},{"osd":2,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":1.149,"15min":1.149},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.443,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.27200000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48399999999999999,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61099999999999999,"15min":0.56499999999999995},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70799999999999996,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.66200000000000003,"15min":0.63200000000000001},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.67700000000000005,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.52900000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77600000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.53900000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67300000000000004,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.58699999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67600000000000005,"15min":0.629},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83199999999999996,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69299999999999995,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9380,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362460,"statfs":{"total":96636764160,"available":96627159040,"internally_reserved":0,"allocated":684032,"data_stored":299123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.45100000000000001,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45500000000000002,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51800000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.311}]},{"osd":2,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":2.032,"15min":2.032},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41099999999999998,"15min":0.42899999999999999},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60099999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.30399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":1.788,"15min":1.788},"last":0.373},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79600000000000004,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":1.002,"15min":1.002},"last":0.61499999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85199999999999998,"5min":1.825,"15min":1.825},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72699999999999998,"5min":1.75,"15min":1.75},"last":0.59399999999999997}]},{"osd":6,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.56399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9644,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362196,"statfs":{"total":96636764160,"available":96626888704,"internally_reserved":0,"allocated":692224,"data_stored":299179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.19,"15min":0.19},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.311},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.314,"15min":0.311},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.311,"15min":0.309},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.40699999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.377}]},{"osd":3,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.35199999999999998,"15min":0.34200000000000003},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.48499999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.34899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.35399999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81000000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.443}]},{"osd":6,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":1.294,"15min":1.294},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79000000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.56200000000000006},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.46200000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81499999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10596,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361244,"statfs":{"total":96636764160,"available":96625913856,"internally_reserved":0,"allocated":1273856,"data_stored":883217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34399999999999997,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":1.319,"15min":1.319},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38900000000000001,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.46999999999999997,"5min":1.915,"15min":1.915},"last":0.435}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.39900000000000002,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.45200000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.50700000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55500000000000005,"5min":1.278,"15min":1.278},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76200000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79400000000000004,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72399999999999998,"5min":1.371,"15min":1.371},"last":0.70699999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76400000000000001,"5min":1.857,"15min":1.857},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78800000000000003,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.71799999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.66000000000000003,"15min":0.67300000000000004},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80900000000000005,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.53100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9820,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94362020,"statfs":{"total":96636764160,"available":96626708480,"internally_reserved":0,"allocated":675840,"data_stored":293255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.91800000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.41099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.008,"5min":1.073,"15min":1.073},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.47799999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71999999999999997,"5min":1.492,"15min":1.492},"last":0.54100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51900000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.378},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39700000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67100000000000004,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.434}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10692,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361148,"statfs":{"total":96636764160,"available":96625815552,"internally_reserved":0,"allocated":1306624,"data_stored":901110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.85999999999999999,"15min":0.97199999999999998},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8069999999999999,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.629},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82699999999999996,"15min":0.95099999999999996},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.90200000000000002},"min":{"1min":0.48599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.718,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.89400000000000002},"min":{"1min":0.42999999999999999,"5min":0.378,"15min":0.378},"max":{"1min":2.8559999999999999,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.90700000000000003},"min":{"1min":0.47899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.1139999999999999,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":4.891,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.89300000000000002,"15min":1.0089999999999999},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.0579999999999998,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75700000000000001,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.54900000000000004,"15min":0.51300000000000001},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.524,"5min":5.524,"15min":5.524},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":0.66500000000000004}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:21:55.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T08:21:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.501+0000 7f977d8d3700 1 Processor -- start 2023-11-08T08:21:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.501+0000 7f977d8d3700 1 -- start start 2023-11-08T08:21:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f977d8d3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f97780b22a0 0x7f97780cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f977d8d3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97780d1ee0 con 0x7f977806ab10 2023-11-08T08:21:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f977d8d3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97780cdb10 con 0x7f97780cd5a0 2023-11-08T08:21:55.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f977d8d3700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97780d1890 con 0x7f97780b22a0 2023-11-08T08:21:55.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f97777fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f97780b22a0 0x7f97780cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f9776ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f977806ab10 0x7f977806aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58546/0 (socket says 172.21.15.116:58546) 2023-11-08T08:21:55.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f9776ffd700 1 -- 172.21.15.116:0/2282710493 learned_addr learned my addr 172.21.15.116:0/2282710493 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:55.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f97777fe700 1 -- 172.21.15.116:0/2282710493 >> v1:172.21.15.129:6789/0 conn(0x7f97780cd5a0 legacy=0x7f97780cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:55.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f97777fe700 1 -- 172.21.15.116:0/2282710493 >> v1:172.21.15.116:6789/0 conn(0x7f977806ab10 legacy=0x7f977806aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f9775ffb700 1 -- 172.21.15.116:0/2282710493 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350991095 0 0) 0x7f97780d1ee0 con 0x7f977806ab10 2023-11-08T08:21:55.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f97777fe700 1 -- 172.21.15.116:0/2282710493 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97780d1b70 con 0x7f97780b22a0 2023-11-08T08:21:55.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f97777fe700 1 --2- 172.21.15.116:0/2282710493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780b22a0 0x7f97780cfb50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9770009d90 tx=0x7f977000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0723e5816f06354 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:55.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.502+0000 7f9775ffb700 1 -- 172.21.15.116:0/2282710493 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977000e070 con 0x7f97780b22a0 2023-11-08T08:21:55.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f9775ffb700 1 -- 172.21.15.116:0/2282710493 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977000bb90 con 0x7f97780b22a0 2023-11-08T08:21:55.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f9775ffb700 1 -- 172.21.15.116:0/2282710493 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770003a20 con 0x7f97780b22a0 2023-11-08T08:21:55.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 -- 172.21.15.116:0/2282710493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780b22a0 msgr2=0x7f97780cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 --2- 172.21.15.116:0/2282710493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780b22a0 0x7f97780cfb50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9770009d90 tx=0x7f977000b330 comp rx=0 tx=0).stop 2023-11-08T08:21:55.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 -- 172.21.15.116:0/2282710493 shutdown_connections 2023-11-08T08:21:55.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 --2- 172.21.15.116:0/2282710493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780b22a0 0x7f97780cfb50 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 -- 172.21.15.116:0/2282710493 >> 172.21.15.116:0/2282710493 conn(0x7f977806c520 msgr2=0x7f977806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:55.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 -- 172.21.15.116:0/2282710493 shutdown_connections 2023-11-08T08:21:55.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.503+0000 7f977d8d3700 1 -- 172.21.15.116:0/2282710493 wait complete. 2023-11-08T08:21:55.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.504+0000 7f977d8d3700 1 Processor -- start 2023-11-08T08:21:55.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.504+0000 7f977d8d3700 1 -- start start 2023-11-08T08:21:55.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f977d8d3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f977806ab10 0x7f97780c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f977d8d3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f97780b22a0 0x7f9778158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f977d8d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 0x7f9778145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f977d8d3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780c8320 con 0x7f97780cd5a0 2023-11-08T08:21:55.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f977d8d3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780c8600 con 0x7f977806ab10 2023-11-08T08:21:55.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f977d8d3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97780c88e0 con 0x7f97780b22a0 2023-11-08T08:21:55.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f97777fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 0x7f9778145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f97777fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 0x7f9778145450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53864/0 (socket says 172.21.15.116:53864) 2023-11-08T08:21:55.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f97777fe700 1 -- 172.21.15.116:0/4215024276 learned_addr learned my addr 172.21.15.116:0/4215024276 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:55.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.505+0000 7f97777fe700 1 -- 172.21.15.116:0/4215024276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f97780b22a0 msgr2=0x7f9778158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f97777fe700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f97780b22a0 0x7f9778158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f97777fe700 1 -- 172.21.15.116:0/4215024276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f977806ab10 msgr2=0x7f97780c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f97777fe700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f977806ab10 0x7f97780c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f97777fe700 1 -- 172.21.15.116:0/4215024276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9770009a70 con 0x7f97780cd5a0 2023-11-08T08:21:55.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f97777fe700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 0x7f9778145450 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f977806d0f0 tx=0x7f9770004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:55.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770012630 con 0x7f97780cd5a0 2023-11-08T08:21:55.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97700127b0 con 0x7f97780cd5a0 2023-11-08T08:21:55.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97780b1ec0 con 0x7f97780cd5a0 2023-11-08T08:21:55.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770012b60 con 0x7f97780cd5a0 2023-11-08T08:21:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.506+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977815cad0 con 0x7f97780cd5a0 2023-11-08T08:21:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.507+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97780b2050 con 0x7f97780cd5a0 2023-11-08T08:21:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.507+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f977002c070 con 0x7f97780cd5a0 2023-11-08T08:21:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.508+0000 7f975ffff700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f97640682f0 0x7f976406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:55.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.508+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f9770087b60 con 0x7f97780cd5a0 2023-11-08T08:21:55.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.508+0000 7f9776ffd700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f97640682f0 0x7f976406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:55.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.508+0000 7f9776ffd700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f97640682f0 0x7f976406a7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f976c0011e0 tx=0x7f976c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:55.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.512+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97700546c0 con 0x7f97780cd5a0 2023-11-08T08:21:55.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.696+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97780743f0 con 0x7f97780cd5a0 2023-11-08T08:21:55.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.697+0000 7f975ffff700 1 -- 172.21.15.116:0/4215024276 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+14389 (secure 0 0 0) 0x7f977002a090 con 0x7f97780cd5a0 2023-11-08T08:21:55.699 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:21:55.700 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":59,"fsid":"2f470591-d661-4b50-b7a5-062dfcf46b1d","created":"2023-11-08T08:18:24.687009+0000","modified":"2023-11-08T08:21:52.590926+0000","last_up_change":"2023-11-08T08:18:35.447743+0000","last_in_change":"2023-11-08T08:18:27.217106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-11-08T08:18:35.994710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-11-08T08:18:36.392792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"028d5009-e114-46ca-86e3-92451886a981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":121157},{"type":"v1","addr":"172.21.15.116:6825","nonce":121157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":121157},{"type":"v1","addr":"172.21.15.116:6827","nonce":121157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":121157},{"type":"v1","addr":"172.21.15.116:6831","nonce":121157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":121157},{"type":"v1","addr":"172.21.15.116:6829","nonce":121157}]},"public_addr":"172.21.15.116:6825/121157","cluster_addr":"172.21.15.116:6827/121157","heartbeat_back_addr":"172.21.15.116:6831/121157","heartbeat_front_addr":"172.21.15.116:6829/121157","state":["exists","up"]},{"osd":1,"uuid":"0a098844-7946-42be-b0b5-0799a4fe3407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":121158},{"type":"v1","addr":"172.21.15.116:6801","nonce":121158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":121158},{"type":"v1","addr":"172.21.15.116:6803","nonce":121158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":121158},{"type":"v1","addr":"172.21.15.116:6807","nonce":121158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":121158},{"type":"v1","addr":"172.21.15.116:6805","nonce":121158}]},"public_addr":"172.21.15.116:6801/121158","cluster_addr":"172.21.15.116:6803/121158","heartbeat_back_addr":"172.21.15.116:6807/121158","heartbeat_front_addr":"172.21.15.116:6805/121158","state":["exists","up"]},{"osd":2,"uuid":"b092ec54-86b1-4d39-bbdc-6cb4d026ad59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":121156},{"type":"v1","addr":"172.21.15.116:6817","nonce":121156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":121156},{"type":"v1","addr":"172.21.15.116:6819","nonce":121156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":121156},{"type":"v1","addr":"172.21.15.116:6823","nonce":121156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":121156},{"type":"v1","addr":"172.21.15.116:6821","nonce":121156}]},"public_addr":"172.21.15.116:6817/121156","cluster_addr":"172.21.15.116:6819/121156","heartbeat_back_addr":"172.21.15.116:6823/121156","heartbeat_front_addr":"172.21.15.116:6821/121156","state":["exists","up"]},{"osd":3,"uuid":"081cf617-173d-4c8e-b5fa-60f71b817a59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":121155},{"type":"v1","addr":"172.21.15.116:6809","nonce":121155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":121155},{"type":"v1","addr":"172.21.15.116:6811","nonce":121155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":121155},{"type":"v1","addr":"172.21.15.116:6815","nonce":121155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":121155},{"type":"v1","addr":"172.21.15.116:6813","nonce":121155}]},"public_addr":"172.21.15.116:6809/121155","cluster_addr":"172.21.15.116:6811/121155","heartbeat_back_addr":"172.21.15.116:6815/121155","heartbeat_front_addr":"172.21.15.116:6813/121155","state":["exists","up"]},{"osd":4,"uuid":"d55405eb-ebeb-432f-9309-8fe8e494663e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":121420},{"type":"v1","addr":"172.21.15.129:6801","nonce":121420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":121420},{"type":"v1","addr":"172.21.15.129:6803","nonce":121420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":121420},{"type":"v1","addr":"172.21.15.129:6807","nonce":121420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":121420},{"type":"v1","addr":"172.21.15.129:6805","nonce":121420}]},"public_addr":"172.21.15.129:6801/121420","cluster_addr":"172.21.15.129:6803/121420","heartbeat_back_addr":"172.21.15.129:6807/121420","heartbeat_front_addr":"172.21.15.129:6805/121420","state":["exists","up"]},{"osd":5,"uuid":"a8ddead2-5aa5-4b7f-b6a0-01a42c452b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":121421},{"type":"v1","addr":"172.21.15.129:6809","nonce":121421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":121421},{"type":"v1","addr":"172.21.15.129:6811","nonce":121421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":121421},{"type":"v1","addr":"172.21.15.129:6815","nonce":121421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":121421},{"type":"v1","addr":"172.21.15.129:6813","nonce":121421}]},"public_addr":"172.21.15.129:6809/121421","cluster_addr":"172.21.15.129:6811/121421","heartbeat_back_addr":"172.21.15.129:6815/121421","heartbeat_front_addr":"172.21.15.129:6813/121421","state":["exists","up"]},{"osd":6,"uuid":"e053dced-1e55-4ff7-ab49-72dd3f415e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":121459},{"type":"v1","addr":"172.21.15.129:6825","nonce":121459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":121459},{"type":"v1","addr":"172.21.15.129:6827","nonce":121459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":121459},{"type":"v1","addr":"172.21.15.129:6831","nonce":121459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":121459},{"type":"v1","addr":"172.21.15.129:6829","nonce":121459}]},"public_addr":"172.21.15.129:6825/121459","cluster_addr":"172.21.15.129:6827/121459","heartbeat_back_addr":"172.21.15.129:6831/121459","heartbeat_front_addr":"172.21.15.129:6829/121459","state":["exists","up"]},{"osd":7,"uuid":"87241c04-a052-4342-80f0-139e223046f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":121422},{"type":"v1","addr":"172.21.15.129:6817","nonce":121422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":121422},{"type":"v1","addr":"172.21.15.129:6819","nonce":121422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":121422},{"type":"v1","addr":"172.21.15.129:6823","nonce":121422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":121422},{"type":"v1","addr":"172.21.15.129:6821","nonce":121422}]},"public_addr":"172.21.15.129:6817/121422","cluster_addr":"172.21.15.129:6819/121422","heartbeat_back_addr":"172.21.15.129:6823/121422","heartbeat_front_addr":"172.21.15.129:6821/121422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.359308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.344924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.354031+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.410220+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.373974+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.325842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.356321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T08:18:32.328184+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/4191005747":"2023-11-09T08:21:49.245410+0000","172.21.15.129:6833/4191005747":"2023-11-09T08:21:49.245410+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T08:21:55.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f97640682f0 msgr2=0x7f976406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f97640682f0 0x7f976406a7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f976c0011e0 tx=0x7f976c0079b0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 msgr2=0x7f9778145450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:55.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 0x7f9778145450 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f977806d0f0 tx=0x7f9770004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 shutdown_connections 2023-11-08T08:21:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97780cd5a0 0x7f9778145450 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f97640682f0 0x7f976406a7a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f977806ab10 0x7f97780c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 --2- 172.21.15.116:0/4215024276 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f97780b22a0 0x7f9778158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 >> 172.21.15.116:0/4215024276 conn(0x7f977806c520 msgr2=0x7f97780b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 shutdown_connections 2023-11-08T08:21:55.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:55.699+0000 7f977d8d3700 1 -- 172.21.15.116:0/4215024276 wait complete. 2023-11-08T08:21:56.713 INFO:tasks.ceph:Scrubbing osd.0 2023-11-08T08:21:56.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.846+0000 7fd0f3fd7700 1 Processor -- start 2023-11-08T08:21:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.846+0000 7fd0f3fd7700 1 -- start start 2023-11-08T08:21:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f3fd7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 0x7fd0ec0af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f3fd7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ec0cf3e0 con 0x7fd0ec161160 2023-11-08T08:21:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f3fd7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ec0cd800 con 0x7fd0ec071320 2023-11-08T08:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f3fd7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0cdae0 con 0x7fd0ec15cc50 2023-11-08T08:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f1572700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd0ec161160 0x7fd0ec070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58560/0 (socket says 172.21.15.116:58560) 2023-11-08T08:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f1572700 1 -- 172.21.15.116:0/3867387843 learned_addr learned my addr 172.21.15.116:0/3867387843 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f1d73700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 0x7fd0ec0af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941666413 0 0) 0x7fd0ec0cf3e0 con 0x7fd0ec161160 2023-11-08T08:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d80035b0 con 0x7fd0ec161160 2023-11-08T08:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492012145 0 0) 0x7fd0ec0cd800 con 0x7fd0ec071320 2023-11-08T08:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ec0cf3e0 con 0x7fd0ec071320 2023-11-08T08:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117594722 0 0) 0x7fd0d80035b0 con 0x7fd0ec161160 2023-11-08T08:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 msgr2=0x7fd0ec0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 0x7fd0ec0af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 >> v1:172.21.15.129:6789/0 conn(0x7fd0ec071320 legacy=0x7fd0ec06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0cdf70 con 0x7fd0ec161160 2023-11-08T08:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7fd0dc001670 con 0x7fd0ec161160 2023-11-08T08:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 >> v1:172.21.15.116:6789/0 conn(0x7fd0ec161160 legacy=0x7fd0ec070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.847+0000 7fd0f0d71700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 0x7fd0d8003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0d8004170 0x7fd0d8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d8004d90 0x7fd0d8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0cdae0 con 0x7fd0d8004d90 2023-11-08T08:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800bb20 con 0x7fd0d8004170 2023-11-08T08:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800bce0 con 0x7fd0ec15cc50 2023-11-08T08:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d8004d90 0x7fd0d8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f1d73700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 0x7fd0d8003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f1572700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0d8004170 0x7fd0d8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 -- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 msgr2=0x7fd0d8003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec15cc50 0x7fd0d8003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 -- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0d8004170 msgr2=0x7fd0d8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0d8004170 0x7fd0d8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 -- 172.21.15.116:0/3867387843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80035b0 con 0x7fd0d8004d90 2023-11-08T08:21:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f2574700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d8004d90 0x7fd0d8005200 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400c4c0 tx=0x7fd0d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a988e6a40fb0dd0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4003a00 con 0x7fd0d8004d90 2023-11-08T08:21:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e4003b60 con 0x7fd0d8004d90 2023-11-08T08:21:56.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.848+0000 7fd0f0d71700 1 -- 172.21.15.116:0/3867387843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40050d0 con 0x7fd0d8004d90 2023-11-08T08:21:56.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/3867387843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d8004d90 msgr2=0x7fd0d8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f3fd7700 1 --2- 172.21.15.116:0/3867387843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d8004d90 0x7fd0d8005200 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400c4c0 tx=0x7fd0d8005880 comp rx=0 tx=0).stop 2023-11-08T08:21:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f2574700 1 -- 172.21.15.116:0/3867387843 reap_dead start 2023-11-08T08:21:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/3867387843 shutdown_connections 2023-11-08T08:21:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/3867387843 >> 172.21.15.116:0/3867387843 conn(0x7fd0ec0cfc30 msgr2=0x7fd0ec0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/3867387843 shutdown_connections 2023-11-08T08:21:56.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.849+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/3867387843 wait complete. 2023-11-08T08:21:56.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.850+0000 7fd0f3fd7700 1 Processor -- start 2023-11-08T08:21:56.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.850+0000 7fd0f3fd7700 1 -- start start 2023-11-08T08:21:56.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f3fd7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec1f5730 0x7fd0ec15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f3fd7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ec15ae40 0x7fd0ec15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f3fd7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0ec15ff70 0x7fd0ec1f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f3fd7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1f37e0 con 0x7fd0ec15ae40 2023-11-08T08:21:56.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f3fd7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1f28a0 con 0x7fd0ec15ff70 2023-11-08T08:21:56.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f3fd7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1f2b60 con 0x7fd0ec1f5730 2023-11-08T08:21:56.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f1572700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ec15ae40 0x7fd0ec15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f1572700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ec15ae40 0x7fd0ec15fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53876/0 (socket says 172.21.15.116:53876) 2023-11-08T08:21:56.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.851+0000 7fd0f1572700 1 -- 172.21.15.116:0/2353439609 learned_addr learned my addr 172.21.15.116:0/2353439609 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:56.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f1572700 1 -- 172.21.15.116:0/2353439609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec1f5730 msgr2=0x7fd0ec15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f1572700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fd0ec1f5730 0x7fd0ec15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f1572700 1 -- 172.21.15.116:0/2353439609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0ec15ff70 msgr2=0x7fd0ec1f5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f1572700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0ec15ff70 0x7fd0ec1f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f1572700 1 -- 172.21.15.116:0/2353439609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0cf610 con 0x7fd0ec15ae40 2023-11-08T08:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f1572700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ec15ae40 0x7fd0ec15fa30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0048d0 tx=0x7fd0dc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc00f5e0 con 0x7fd0ec15ae40 2023-11-08T08:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc00f740 con 0x7fd0ec15ae40 2023-11-08T08:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec1616b0 con 0x7fd0ec15ae40 2023-11-08T08:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc00faf0 con 0x7fd0ec15ae40 2023-11-08T08:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.852+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec15d020 con 0x7fd0ec15ae40 2023-11-08T08:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.853+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0ec0743f0 con 0x7fd0ec15ae40 2023-11-08T08:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.853+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd0dc00fc50 con 0x7fd0ec15ae40 2023-11-08T08:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0e27fc700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0d00682f0 0x7fd0d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fd0dc086510 con 0x7fd0ec15ae40 2023-11-08T08:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0f1d73700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0d00682f0 0x7fd0d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0e27fc700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7fd0d006e760 0x7fd0d0070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 --> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0d0071240 con 0x7fd0d006e760 2023-11-08T08:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0dc086950 con 0x7fd0ec15ae40 2023-11-08T08:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0f2574700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7fd0d006e760 0x7fd0d0070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0f1d73700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0d00682f0 0x7fd0d006a7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec15ba20 tx=0x7fd0e8009670 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.854+0000 7fd0f2574700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7fd0d006e760 0x7fd0d0070b70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.855+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== osd.0 v2:172.21.15.116:6824/121157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd0d0071240 con 0x7fd0d006e760 2023-11-08T08:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 --> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd0ec0c6450 con 0x7fd0d006e760 2023-11-08T08:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0e27fc700 1 -- 172.21.15.116:0/2353439609 <== osd.0 v2:172.21.15.116:6824/121157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd0ec0c6450 con 0x7fd0d006e760 2023-11-08T08:21:56.879 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:56.880 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7fd0d006e760 msgr2=0x7fd0d0070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6824/121157,v1:172.21.15.116:6825/121157] conn(0x7fd0d006e760 0x7fd0d0070b70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0d00682f0 msgr2=0x7fd0d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fd0d00682f0 0x7fd0d006a7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec15ba20 tx=0x7fd0e8009670 comp rx=0 tx=0).stop 2023-11-08T08:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ec15ae40 msgr2=0x7fd0ec15fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 --2- 172.21.15.116:0/2353439609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ec15ae40 0x7fd0ec15fa30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0048d0 tx=0x7fd0dc007620 comp rx=0 tx=0).stop 2023-11-08T08:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f2574700 1 -- 172.21.15.116:0/2353439609 reap_dead start 2023-11-08T08:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 shutdown_connections 2023-11-08T08:21:56.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 >> 172.21.15.116:0/2353439609 conn(0x7fd0ec0cfc30 msgr2=0x7fd0ec06e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:56.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 shutdown_connections 2023-11-08T08:21:56.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:56.869+0000 7fd0f3fd7700 1 -- 172.21.15.116:0/2353439609 wait complete. 2023-11-08T08:21:56.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-08T08:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.004+0000 7f4cfeae3700 1 Processor -- start 2023-11-08T08:21:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.004+0000 7f4cfeae3700 1 -- start start 2023-11-08T08:21:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfeae3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cf815c8a0 0x7f4cf806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfeae3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf815cc70 con 0x7f4cf8071220 2023-11-08T08:21:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfeae3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf806c880 con 0x7f4cf80ce740 2023-11-08T08:21:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfeae3700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf80b2470 con 0x7f4cf815c8a0 2023-11-08T08:21:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfc87f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cf815c8a0 0x7f4cf806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfd080700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4cf8071220 0x7f4cf80b1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58574/0 (socket says 172.21.15.116:58574) 2023-11-08T08:21:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.005+0000 7f4cfd080700 1 -- 172.21.15.116:0/2614580749 learned_addr learned my addr 172.21.15.116:0/2614580749 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfc87f700 1 -- 172.21.15.116:0/2614580749 >> v1:172.21.15.129:6789/0 conn(0x7f4cf80ce740 legacy=0x7f4cf80b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfc87f700 1 -- 172.21.15.116:0/2614580749 >> v1:172.21.15.116:6789/0 conn(0x7f4cf8071220 legacy=0x7f4cf80b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfc87f700 1 -- 172.21.15.116:0/2614580749 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf806cad0 con 0x7f4cf815c8a0 2023-11-08T08:21:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cf77fe700 1 -- 172.21.15.116:0/2614580749 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493299150 0 0) 0x7f4cf815cc70 con 0x7f4cf8071220 2023-11-08T08:21:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfc87f700 1 --2- 172.21.15.116:0/2614580749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf815c8a0 0x7f4cf806a270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009840 tx=0x7f4cec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefa292ed7665962 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cf77fe700 1 -- 172.21.15.116:0/2614580749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec004230 con 0x7f4cf815c8a0 2023-11-08T08:21:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cf77fe700 1 -- 172.21.15.116:0/2614580749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec004390 con 0x7f4cf815c8a0 2023-11-08T08:21:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cf77fe700 1 -- 172.21.15.116:0/2614580749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec0036a0 con 0x7f4cf815c8a0 2023-11-08T08:21:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 -- 172.21.15.116:0/2614580749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf815c8a0 msgr2=0x7f4cf806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/2614580749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf815c8a0 0x7f4cf806a270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009840 tx=0x7f4cec009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:57.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 -- 172.21.15.116:0/2614580749 shutdown_connections 2023-11-08T08:21:57.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/2614580749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf815c8a0 0x7f4cf806a270 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 -- 172.21.15.116:0/2614580749 >> 172.21.15.116:0/2614580749 conn(0x7f4cf806c010 msgr2=0x7f4cf80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 -- 172.21.15.116:0/2614580749 shutdown_connections 2023-11-08T08:21:57.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.006+0000 7f4cfeae3700 1 -- 172.21.15.116:0/2614580749 wait complete. 2023-11-08T08:21:57.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 Processor -- start 2023-11-08T08:21:57.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 -- start start 2023-11-08T08:21:57.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 0x7f4cf815a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4cf80ce740 0x7f4cf8158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cf815c8a0 0x7f4cf8158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf815b040 con 0x7f4cf8071220 2023-11-08T08:21:57.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf815b320 con 0x7f4cf815c8a0 2023-11-08T08:21:57.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.008+0000 7f4cfeae3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf815b600 con 0x7f4cf80ce740 2023-11-08T08:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 0x7f4cf815a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 0x7f4cf815a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53888/0 (socket says 172.21.15.116:53888) 2023-11-08T08:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 -- 172.21.15.116:0/1448696315 learned_addr learned my addr 172.21.15.116:0/1448696315 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfd080700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cf815c8a0 0x7f4cf8158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cf7fff700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4cf80ce740 0x7f4cf8158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 -- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4cf80ce740 msgr2=0x7f4cf8158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4cf80ce740 0x7f4cf8158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 -- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cf815c8a0 msgr2=0x7f4cf8158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cf815c8a0 0x7f4cf8158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 -- 172.21.15.116:0/1448696315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0094a0 con 0x7f4cf8071220 2023-11-08T08:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfd080700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cf815c8a0 0x7f4cf8158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfc87f700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 0x7f4cf815a5f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0090b0 tx=0x7f4cec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec011630 con 0x7f4cf8071220 2023-11-08T08:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec011790 con 0x7f4cf8071220 2023-11-08T08:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec011b40 con 0x7f4cf8071220 2023-11-08T08:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf806d090 con 0x7f4cf8071220 2023-11-08T08:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.009+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf806cdb0 con 0x7f4cf8071220 2023-11-08T08:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.010+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf806d370 con 0x7f4cf8071220 2023-11-08T08:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.010+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4cec017020 con 0x7f4cf8071220 2023-11-08T08:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.011+0000 7f4cf57fa700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4cdc0682f0 0x7f4cdc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.011+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f4cec087220 con 0x7f4cf8071220 2023-11-08T08:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.011+0000 7f4cf7fff700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4cdc0682f0 0x7f4cdc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.011+0000 7f4cf7fff700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4cdc0682f0 0x7f4cdc06a7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4cf81f5980 tx=0x7f4ce4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.015+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4cec053d40 con 0x7f4cf8071220 2023-11-08T08:21:57.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.196+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cf815b490 con 0x7f4cdc0682f0 2023-11-08T08:21:57.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.196+0000 7f4cf57fa700 1 -- 172.21.15.116:0/1448696315 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4cf815b490 con 0x7f4cdc0682f0 2023-11-08T08:21:57.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.198+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4cdc0682f0 msgr2=0x7f4cdc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.198+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4cdc0682f0 0x7f4cdc06a7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4cf81f5980 tx=0x7f4ce4009040 comp rx=0 tx=0).stop 2023-11-08T08:21:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.198+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 msgr2=0x7f4cf815a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.198+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 0x7f4cf815a5f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0090b0 tx=0x7f4cec0047c0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 shutdown_connections 2023-11-08T08:21:57.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f4cdc0682f0 0x7f4cdc06a7a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cf815c8a0 0x7f4cf8158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf8071220 0x7f4cf815a5f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 --2- 172.21.15.116:0/1448696315 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4cf80ce740 0x7f4cf8158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 >> 172.21.15.116:0/1448696315 conn(0x7f4cf806c010 msgr2=0x7f4cf80cc880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 shutdown_connections 2023-11-08T08:21:57.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.199+0000 7f4cfeae3700 1 -- 172.21.15.116:0/1448696315 wait complete. 2023-11-08T08:21:57.204 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 0 to deep-scrub 2023-11-08T08:21:57.211 INFO:tasks.ceph:Scrubbing osd.1 2023-11-08T08:21:57.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.334+0000 7f3284798700 1 Processor -- start 2023-11-08T08:21:57.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.334+0000 7f3284798700 1 -- start start 2023-11-08T08:21:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3284798700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f327c15c8f0 0x7f327c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3284798700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f327c0ccef0 0x7f327c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3284798700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0cbe50 con 0x7f327c0d0cb0 2023-11-08T08:21:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3284798700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c0cc130 con 0x7f327c0ccef0 2023-11-08T08:21:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3284798700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c0cc410 con 0x7f327c15c8f0 2023-11-08T08:21:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3282d35700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f327c0ccef0 0x7f327c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3282534700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f327c0d0cb0 0x7f327c0d1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58578/0 (socket says 172.21.15.116:58578) 2023-11-08T08:21:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3282534700 1 -- 172.21.15.116:0/2093993658 learned_addr learned my addr 172.21.15.116:0/2093993658 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3282d35700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f327c0ccef0 0x7f327c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53894/0 (socket says 172.21.15.116:53894) 2023-11-08T08:21:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.335+0000 7f3281d33700 1 --2- 172.21.15.116:0/2093993658 >> v2:172.21.15.129:3300/0 conn(0x7f327c15c8f0 0x7f327c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3282d35700 1 -- 172.21.15.116:0/2093993658 >> v2:172.21.15.129:3300/0 conn(0x7f327c15c8f0 msgr2=0x7f327c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3282d35700 1 --2- 172.21.15.116:0/2093993658 >> v2:172.21.15.129:3300/0 conn(0x7f327c15c8f0 0x7f327c06b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3282d35700 1 -- 172.21.15.116:0/2093993658 >> v1:172.21.15.116:6789/0 conn(0x7f327c0d0cb0 legacy=0x7f327c0d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3282d35700 1 -- 172.21.15.116:0/2093993658 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c0cc6f0 con 0x7f327c0ccef0 2023-11-08T08:21:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3281532700 1 -- 172.21.15.116:0/2093993658 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147389230 0 0) 0x7f327c0cbe50 con 0x7f327c0d0cb0 2023-11-08T08:21:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3281d33700 1 --2- 172.21.15.116:0/2093993658 >> v2:172.21.15.129:3300/0 conn(0x7f327c15c8f0 0x7f327c06b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3282d35700 1 --2- 172.21.15.116:0/2093993658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c06b8c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f327c06b040 tx=0x7f327400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9695dcf237645b9b server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3281532700 1 -- 172.21.15.116:0/2093993658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400e070 con 0x7f327c0ccef0 2023-11-08T08:21:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3281532700 1 -- 172.21.15.116:0/2093993658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327400bbd0 con 0x7f327c0ccef0 2023-11-08T08:21:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3281532700 1 -- 172.21.15.116:0/2093993658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274012430 con 0x7f327c0ccef0 2023-11-08T08:21:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3284798700 1 -- 172.21.15.116:0/2093993658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 msgr2=0x7f327c06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.336+0000 7f3284798700 1 --2- 172.21.15.116:0/2093993658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c06b8c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f327c06b040 tx=0x7f327400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.337+0000 7f3284798700 1 -- 172.21.15.116:0/2093993658 shutdown_connections 2023-11-08T08:21:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.337+0000 7f3284798700 1 --2- 172.21.15.116:0/2093993658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c06b8c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.337+0000 7f3284798700 1 --2- 172.21.15.116:0/2093993658 >> v2:172.21.15.129:3300/0 conn(0x7f327c15c8f0 0x7f327c06b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.337+0000 7f3284798700 1 -- 172.21.15.116:0/2093993658 >> 172.21.15.116:0/2093993658 conn(0x7f327c06ab90 msgr2=0x7f327c099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.337+0000 7f3284798700 1 -- 172.21.15.116:0/2093993658 shutdown_connections 2023-11-08T08:21:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.337+0000 7f3284798700 1 -- 172.21.15.116:0/2093993658 wait complete. 2023-11-08T08:21:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 Processor -- start 2023-11-08T08:21:57.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 -- start start 2023-11-08T08:21:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c0b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f327c0d0cb0 0x7f327c1f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f327c15c8f0 0x7f327c0cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c157b90 con 0x7f327c0ccef0 2023-11-08T08:21:57.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c0c7e20 con 0x7f327c15c8f0 2023-11-08T08:21:57.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3284798700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c1f2540 con 0x7f327c0d0cb0 2023-11-08T08:21:57.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3282534700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c0b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.338+0000 7f3282534700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c0b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53900/0 (socket says 172.21.15.116:53900) 2023-11-08T08:21:57.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 -- 172.21.15.116:0/105165947 learned_addr learned my addr 172.21.15.116:0/105165947 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 -- 172.21.15.116:0/105165947 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f327c0d0cb0 msgr2=0x7f327c1f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f327c0d0cb0 0x7f327c1f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 -- 172.21.15.116:0/105165947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f327c15c8f0 msgr2=0x7f327c0cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f327c15c8f0 0x7f327c0cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 -- 172.21.15.116:0/105165947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274009a70 con 0x7f327c0ccef0 2023-11-08T08:21:57.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3282534700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c0b1870 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f327800c700 tx=0x7f327800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780074e0 con 0x7f327c0ccef0 2023-11-08T08:21:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278007640 con 0x7f327c0ccef0 2023-11-08T08:21:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278005a30 con 0x7f327c0ccef0 2023-11-08T08:21:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c06be00 con 0x7f327c0ccef0 2023-11-08T08:21:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.339+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c0cc9d0 con 0x7f327c0ccef0 2023-11-08T08:21:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.340+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f327c0ad980 con 0x7f327c0ccef0 2023-11-08T08:21:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.340+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3278019030 con 0x7f327c0ccef0 2023-11-08T08:21:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3272ffd700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32600682f0 0x7f326006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f3278084d10 con 0x7f327c0ccef0 2023-11-08T08:21:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3281d33700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32600682f0 0x7f326006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3272ffd700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f326006e760 0x7f3260070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 --> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3260071240 con 0x7f326006e760 2023-11-08T08:21:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3278089030 con 0x7f327c0ccef0 2023-11-08T08:21:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3282d35700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f326006e760 0x7f3260070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3281d33700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32600682f0 0x7f326006a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f327c0b12d0 tx=0x7f326c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.341+0000 7f3282d35700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f326006e760 0x7f3260070b70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.342+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== osd.1 v2:172.21.15.116:6800/121158 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3260071240 con 0x7f326006e760 2023-11-08T08:21:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 --> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f327c0743f0 con 0x7f326006e760 2023-11-08T08:21:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3272ffd700 1 -- 172.21.15.116:0/105165947 <== osd.1 v2:172.21.15.116:6800/121158 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f327c0743f0 con 0x7f326006e760 2023-11-08T08:21:57.361 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:57.361 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:57.362 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f326006e760 msgr2=0x7f3260070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6800/121158,v1:172.21.15.116:6801/121158] conn(0x7f326006e760 0x7f3260070b70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32600682f0 msgr2=0x7f326006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f32600682f0 0x7f326006a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f327c0b12d0 tx=0x7f326c008040 comp rx=0 tx=0).stop 2023-11-08T08:21:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 msgr2=0x7f327c0b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 --2- 172.21.15.116:0/105165947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c0ccef0 0x7f327c0b1870 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f327800c700 tx=0x7f327800cac0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3282d35700 1 -- 172.21.15.116:0/105165947 reap_dead start 2023-11-08T08:21:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 shutdown_connections 2023-11-08T08:21:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.356+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 >> 172.21.15.116:0/105165947 conn(0x7f327c06ab90 msgr2=0x7f327c15aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.357+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 shutdown_connections 2023-11-08T08:21:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.357+0000 7f3284798700 1 -- 172.21.15.116:0/105165947 wait complete. 2023-11-08T08:21:57.369 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-08T08:21:57.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.490+0000 7f17dd1ff700 1 Processor -- start 2023-11-08T08:21:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.490+0000 7f17dd1ff700 1 -- start start 2023-11-08T08:21:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17dd1ff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17d80cf710 0x7f17d80ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17dd1ff700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17d8071890 0x7f17d8071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17dd1ff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d815cc10 con 0x7f17d806a270 2023-11-08T08:21:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17dd1ff700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d806c9b0 con 0x7f17d80cf710 2023-11-08T08:21:57.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17dd1ff700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d806cc90 con 0x7f17d8071890 2023-11-08T08:21:57.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17d80cf710 0x7f17d80ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17d80cf710 0x7f17d80ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53906/0 (socket says 172.21.15.116:53906) 2023-11-08T08:21:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 -- 172.21.15.116:0/2521633300 learned_addr learned my addr 172.21.15.116:0/2521633300 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 -- 172.21.15.116:0/2521633300 >> v2:172.21.15.129:3300/0 conn(0x7f17d8071890 msgr2=0x7f17d8071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d659c700 1 --2- 172.21.15.116:0/2521633300 >> v2:172.21.15.129:3300/0 conn(0x7f17d8071890 0x7f17d8071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 --2- 172.21.15.116:0/2521633300 >> v2:172.21.15.129:3300/0 conn(0x7f17d8071890 0x7f17d8071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 -- 172.21.15.116:0/2521633300 >> v1:172.21.15.129:6789/0 conn(0x7f17d806a270 legacy=0x7f17d80b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 -- 172.21.15.116:0/2521633300 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d806cf70 con 0x7f17d80cf710 2023-11-08T08:21:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.491+0000 7f17d6d9d700 1 --2- 172.21.15.116:0/2521633300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d80cf710 0x7f17d80ce740 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f17c80098f0 tx=0x7f17c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9d6a7068fab0c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17d5d9b700 1 -- 172.21.15.116:0/2521633300 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8004230 con 0x7f17d80cf710 2023-11-08T08:21:57.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17d5d9b700 1 -- 172.21.15.116:0/2521633300 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c8004390 con 0x7f17d80cf710 2023-11-08T08:21:57.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17d5d9b700 1 -- 172.21.15.116:0/2521633300 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c80036a0 con 0x7f17d80cf710 2023-11-08T08:21:57.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 -- 172.21.15.116:0/2521633300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d80cf710 msgr2=0x7f17d80ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/2521633300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d80cf710 0x7f17d80ce740 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f17c80098f0 tx=0x7f17c8009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:57.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 -- 172.21.15.116:0/2521633300 shutdown_connections 2023-11-08T08:21:57.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/2521633300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d80cf710 0x7f17d80ce740 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/2521633300 >> v2:172.21.15.129:3300/0 conn(0x7f17d8071890 0x7f17d8071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 -- 172.21.15.116:0/2521633300 >> 172.21.15.116:0/2521633300 conn(0x7f17d806c010 msgr2=0x7f17d80cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 -- 172.21.15.116:0/2521633300 shutdown_connections 2023-11-08T08:21:57.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.492+0000 7f17dd1ff700 1 -- 172.21.15.116:0/2521633300 wait complete. 2023-11-08T08:21:57.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 Processor -- start 2023-11-08T08:21:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 -- start start 2023-11-08T08:21:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 0x7f17d8156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d8071890 0x7f17d8156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80cf710 0x7f17d81573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8155920 con 0x7f17d806a270 2023-11-08T08:21:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d81526c0 con 0x7f17d8071890 2023-11-08T08:21:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17dd1ff700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8151a40 con 0x7f17d80cf710 2023-11-08T08:21:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17d6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 0x7f17d8156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17d6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 0x7f17d8156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53916/0 (socket says 172.21.15.116:53916) 2023-11-08T08:21:57.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.494+0000 7f17d6d9d700 1 -- 172.21.15.116:0/1225339025 learned_addr learned my addr 172.21.15.116:0/1225339025 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d759e700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80cf710 0x7f17d81573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d6d9d700 1 -- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80cf710 msgr2=0x7f17d81573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d659c700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d8071890 0x7f17d8156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d6d9d700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80cf710 0x7f17d81573c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d6d9d700 1 -- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d8071890 msgr2=0x7f17d8156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d6d9d700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d8071890 0x7f17d8156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d6d9d700 1 -- 172.21.15.116:0/1225339025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8009550 con 0x7f17d806a270 2023-11-08T08:21:57.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d659c700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d8071890 0x7f17d8156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:57.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17d6d9d700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 0x7f17d8156940 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f17c8000c00 tx=0x7f17c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8011830 con 0x7f17d806a270 2023-11-08T08:21:57.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c8011990 con 0x7f17d806a270 2023-11-08T08:21:57.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8011d40 con 0x7f17d806a270 2023-11-08T08:21:57.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d806d530 con 0x7f17d806a270 2023-11-08T08:21:57.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.495+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d806d250 con 0x7f17d806a270 2023-11-08T08:21:57.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.496+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d806a6b0 con 0x7f17d806a270 2023-11-08T08:21:57.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.496+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f17c8017020 con 0x7f17d806a270 2023-11-08T08:21:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.497+0000 7f17cf7fe700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b80682f0 0x7f17b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.497+0000 7f17d659c700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b80682f0 0x7f17b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.497+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f17c8087210 con 0x7f17d806a270 2023-11-08T08:21:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.497+0000 7f17d659c700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b80682f0 0x7f17b806a7a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f17c0006fd0 tx=0x7f17c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.501+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17c8053d70 con 0x7f17d806a270 2023-11-08T08:21:57.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.682+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f17d80743f0 con 0x7f17b80682f0 2023-11-08T08:21:57.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.682+0000 7f17cf7fe700 1 -- 172.21.15.116:0/1225339025 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f17d80743f0 con 0x7f17b80682f0 2023-11-08T08:21:57.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b80682f0 msgr2=0x7f17b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b80682f0 0x7f17b806a7a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f17c0006fd0 tx=0x7f17c0008040 comp rx=0 tx=0).stop 2023-11-08T08:21:57.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 msgr2=0x7f17d8156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 0x7f17d8156940 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f17c8000c00 tx=0x7f17c8004750 comp rx=0 tx=0).stop 2023-11-08T08:21:57.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 shutdown_connections 2023-11-08T08:21:57.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17d80cf710 0x7f17d81573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17d806a270 0x7f17d8156940 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f17b80682f0 0x7f17b806a7a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 --2- 172.21.15.116:0/1225339025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d8071890 0x7f17d8156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 >> 172.21.15.116:0/1225339025 conn(0x7f17d806c010 msgr2=0x7f17d80cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 shutdown_connections 2023-11-08T08:21:57.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.684+0000 7f17dd1ff700 1 -- 172.21.15.116:0/1225339025 wait complete. 2023-11-08T08:21:57.689 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 1 to deep-scrub 2023-11-08T08:21:57.697 INFO:tasks.ceph:Scrubbing osd.2 2023-11-08T08:21:57.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:57.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 Processor -- start 2023-11-08T08:21:57.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 -- start start 2023-11-08T08:21:57.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fb815cce0 0x7f9fb80b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9fb80b1c90 0x7f9fb80b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb80b32b0 con 0x7f9fb80b22e0 2023-11-08T08:21:57.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb80b26e0 con 0x7f9fb815cce0 2023-11-08T08:21:57.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fbd6ec700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb80b2980 con 0x7f9fb80b1c90 2023-11-08T08:21:57.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.821+0000 7f9fb6ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fb815cce0 0x7f9fb80b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fb815cce0 0x7f9fb80b1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53918/0 (socket says 172.21.15.116:53918) 2023-11-08T08:21:57.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/2001888720 learned_addr learned my addr 172.21.15.116:0/2001888720 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/2001888720 >> v2:172.21.15.129:3300/0 conn(0x7f9fb80b1c90 msgr2=0x7f9fb80b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb77fe700 1 --2- 172.21.15.116:0/2001888720 >> v2:172.21.15.129:3300/0 conn(0x7f9fb80b1c90 0x7f9fb80b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 --2- 172.21.15.116:0/2001888720 >> v2:172.21.15.129:3300/0 conn(0x7f9fb80b1c90 0x7f9fb80b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/2001888720 >> v1:172.21.15.129:6789/0 conn(0x7f9fb80b22e0 legacy=0x7f9fb80d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/2001888720 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb80b2c60 con 0x7f9fb815cce0 2023-11-08T08:21:57.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb6ffd700 1 --2- 172.21.15.116:0/2001888720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb815cce0 0x7f9fb80b1750 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9fb806a6f0 tx=0x7f9fac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b5ce1cdb07d842 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb5ffb700 1 -- 172.21.15.116:0/2001888720 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac004230 con 0x7f9fb815cce0 2023-11-08T08:21:57.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb5ffb700 1 -- 172.21.15.116:0/2001888720 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac004390 con 0x7f9fb815cce0 2023-11-08T08:21:57.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.822+0000 7f9fb5ffb700 1 -- 172.21.15.116:0/2001888720 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0036a0 con 0x7f9fb815cce0 2023-11-08T08:21:57.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/2001888720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb815cce0 msgr2=0x7f9fb80b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 --2- 172.21.15.116:0/2001888720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb815cce0 0x7f9fb80b1750 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9fb806a6f0 tx=0x7f9fac009bd0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/2001888720 shutdown_connections 2023-11-08T08:21:57.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 --2- 172.21.15.116:0/2001888720 >> v2:172.21.15.129:3300/0 conn(0x7f9fb80b1c90 0x7f9fb80b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 --2- 172.21.15.116:0/2001888720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb815cce0 0x7f9fb80b1750 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/2001888720 >> 172.21.15.116:0/2001888720 conn(0x7f9fb809c4d0 msgr2=0x7f9fb80d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/2001888720 shutdown_connections 2023-11-08T08:21:57.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.823+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/2001888720 wait complete. 2023-11-08T08:21:57.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.824+0000 7f9fbd6ec700 1 Processor -- start 2023-11-08T08:21:57.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.824+0000 7f9fbd6ec700 1 -- start start 2023-11-08T08:21:57.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb80b1c90 0x7f9fb80b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fb815cce0 0x7f9fb80c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9fb815fb40 0x7f9fb815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb80b8f30 con 0x7f9fb80b1c90 2023-11-08T08:21:57.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb80c7a30 con 0x7f9fb815cce0 2023-11-08T08:21:57.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb80c7d10 con 0x7f9fb815fb40 2023-11-08T08:21:57.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb80b1c90 0x7f9fb80b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb80b1c90 0x7f9fb80b89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53922/0 (socket says 172.21.15.116:53922) 2023-11-08T08:21:57.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/497384903 learned_addr learned my addr 172.21.15.116:0/497384903 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb67fc700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fb815cce0 0x7f9fb80c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb77fe700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9fb815fb40 0x7f9fb815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9fb815fb40 msgr2=0x7f9fb815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9fb815fb40 0x7f9fb815ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fb815cce0 msgr2=0x7f9fb80c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fb815cce0 0x7f9fb80c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 -- 172.21.15.116:0/497384903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac009550 con 0x7f9fb80b1c90 2023-11-08T08:21:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb77fe700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9fb815fb40 0x7f9fb815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fb6ffd700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb80b1c90 0x7f9fb80b89f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9fb808b500 tx=0x7f9fac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac003d80 con 0x7f9fb80b1c90 2023-11-08T08:21:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac01f070 con 0x7f9fb80b1c90 2023-11-08T08:21:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb80d1440 con 0x7f9fb80b1c90 2023-11-08T08:21:57.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac011870 con 0x7f9fb80b1c90 2023-11-08T08:21:57.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.825+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb80b2f40 con 0x7f9fb80b1c90 2023-11-08T08:21:57.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.826+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9fb80d15d0 con 0x7f9fb80b1c90 2023-11-08T08:21:57.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9fac017020 con 0x7f9fb80b1c90 2023-11-08T08:21:57.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9f9f7fe700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9fa40682f0 0x7f9fa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f9fac0872b0 con 0x7f9fb80b1c90 2023-11-08T08:21:57.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9fb67fc700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9fa40682f0 0x7f9fa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9f9f7fe700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f9fa406e760 0x7f9fa4070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 --> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fa4071240 con 0x7f9fa406e760 2023-11-08T08:21:57.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fac089890 con 0x7f9fb80b1c90 2023-11-08T08:21:57.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.827+0000 7f9fb77fe700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f9fa406e760 0x7f9fa4070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.828+0000 7f9fb67fc700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9fa40682f0 0x7f9fa406a7a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9fb814d840 tx=0x7f9fa8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.828+0000 7f9fb77fe700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f9fa406e760 0x7f9fa4070b70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.828+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== osd.2 v2:172.21.15.116:6816/121156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9fa4071240 con 0x7f9fa406e760 2023-11-08T08:21:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.842+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 --> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9fb80743f0 con 0x7f9fa406e760 2023-11-08T08:21:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9f9f7fe700 1 -- 172.21.15.116:0/497384903 <== osd.2 v2:172.21.15.116:6816/121156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9fb80743f0 con 0x7f9fa406e760 2023-11-08T08:21:57.847 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:57.847 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:57.848 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:57.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f9fa406e760 msgr2=0x7f9fa4070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6816/121156,v1:172.21.15.116:6817/121156] conn(0x7f9fa406e760 0x7f9fa4070b70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9fa40682f0 msgr2=0x7f9fa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f9fa40682f0 0x7f9fa406a7a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9fb814d840 tx=0x7f9fa8009040 comp rx=0 tx=0).stop 2023-11-08T08:21:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb80b1c90 msgr2=0x7f9fb80b89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 --2- 172.21.15.116:0/497384903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fb80b1c90 0x7f9fb80b89f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9fb808b500 tx=0x7f9fac004750 comp rx=0 tx=0).stop 2023-11-08T08:21:57.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fb77fe700 1 -- 172.21.15.116:0/497384903 reap_dead start 2023-11-08T08:21:57.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 shutdown_connections 2023-11-08T08:21:57.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 >> 172.21.15.116:0/497384903 conn(0x7f9fb809c4d0 msgr2=0x7f9fb80b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 shutdown_connections 2023-11-08T08:21:57.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.843+0000 7f9fbd6ec700 1 -- 172.21.15.116:0/497384903 wait complete. 2023-11-08T08:21:57.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-08T08:21:57.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 Processor -- start 2023-11-08T08:21:57.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 -- start start 2023-11-08T08:21:57.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff0c0b1650 0x7eff0c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0af0a0 0x7eff0c15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c159ec0 con 0x7eff0c1599b0 2023-11-08T08:21:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0c0b1f60 con 0x7eff0c0b1650 2023-11-08T08:21:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.976+0000 7eff12e91700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c15d8a0 con 0x7eff0c0af0a0 2023-11-08T08:21:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff0c0b1650 0x7eff0c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff0c0b1650 0x7eff0c0b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53932/0 (socket says 172.21.15.116:53932) 2023-11-08T08:21:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 -- 172.21.15.116:0/1043946987 learned_addr learned my addr 172.21.15.116:0/1043946987 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff1142e700 1 --2- 172.21.15.116:0/1043946987 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0af0a0 0x7eff0c15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 -- 172.21.15.116:0/1043946987 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0af0a0 msgr2=0x7eff0c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 --2- 172.21.15.116:0/1043946987 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0af0a0 0x7eff0c15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 -- 172.21.15.116:0/1043946987 >> v1:172.21.15.129:6789/0 conn(0x7eff0c1599b0 legacy=0x7eff0c15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 -- 172.21.15.116:0/1043946987 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c15db80 con 0x7eff0c0b1650 2023-11-08T08:21:57.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff10c2d700 1 --2- 172.21.15.116:0/1043946987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0b1650 0x7eff0c0b1a20 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7eff000098f0 tx=0x7eff00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4c9cceec6716cb server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff0b7fe700 1 -- 172.21.15.116:0/1043946987 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00011830 con 0x7eff0c0b1650 2023-11-08T08:21:57.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff0b7fe700 1 -- 172.21.15.116:0/1043946987 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff00011990 con 0x7eff0c0b1650 2023-11-08T08:21:57.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.977+0000 7eff0b7fe700 1 -- 172.21.15.116:0/1043946987 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00011d20 con 0x7eff0c0b1650 2023-11-08T08:21:57.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 -- 172.21.15.116:0/1043946987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0b1650 msgr2=0x7eff0c0b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 --2- 172.21.15.116:0/1043946987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0b1650 0x7eff0c0b1a20 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7eff000098f0 tx=0x7eff00009c00 comp rx=0 tx=0).stop 2023-11-08T08:21:57.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 -- 172.21.15.116:0/1043946987 shutdown_connections 2023-11-08T08:21:57.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 --2- 172.21.15.116:0/1043946987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0b1650 0x7eff0c0b1a20 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 --2- 172.21.15.116:0/1043946987 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0af0a0 0x7eff0c15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 -- 172.21.15.116:0/1043946987 >> 172.21.15.116:0/1043946987 conn(0x7eff0c06c010 msgr2=0x7eff0c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 -- 172.21.15.116:0/1043946987 shutdown_connections 2023-11-08T08:21:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.978+0000 7eff12e91700 1 -- 172.21.15.116:0/1043946987 wait complete. 2023-11-08T08:21:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.979+0000 7eff12e91700 1 Processor -- start 2023-11-08T08:21:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.979+0000 7eff12e91700 1 -- start start 2023-11-08T08:21:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff0c0af0a0 0x7eff0c14e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0b1650 0x7eff0c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 0x7eff0c0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c0c95f0 con 0x7eff0c0c9150 2023-11-08T08:21:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c0c47f0 con 0x7eff0c0af0a0 2023-11-08T08:21:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c0c4ad0 con 0x7eff0c0b1650 2023-11-08T08:21:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 0x7eff0c0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 0x7eff0c0c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53936/0 (socket says 172.21.15.116:53936) 2023-11-08T08:21:57.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 -- 172.21.15.116:0/4054867147 learned_addr learned my addr 172.21.15.116:0/4054867147 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:57.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 -- 172.21.15.116:0/4054867147 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0b1650 msgr2=0x7eff0c0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0b1650 0x7eff0c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 -- 172.21.15.116:0/4054867147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff0c0af0a0 msgr2=0x7eff0c14e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:57.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff0c0af0a0 0x7eff0c14e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:57.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 -- 172.21.15.116:0/4054867147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff00009550 con 0x7eff0c0c9150 2023-11-08T08:21:57.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff1142e700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 0x7eff0c0c7b80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7eff04009fb0 tx=0x7eff0400e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff04012430 con 0x7eff0c0c9150 2023-11-08T08:21:57.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff04012590 con 0x7eff0c0c9150 2023-11-08T08:21:57.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c15a890 con 0x7eff0c0c9150 2023-11-08T08:21:57.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff04012940 con 0x7eff0c0c9150 2023-11-08T08:21:57.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.980+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c15de60 con 0x7eff0c0c9150 2023-11-08T08:21:57.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.981+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c15aa20 con 0x7eff0c0c9150 2023-11-08T08:21:57.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.982+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff04012aa0 con 0x7eff0c0c9150 2023-11-08T08:21:57.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.982+0000 7eff097fa700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efef00682f0 0x7efef006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:57.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.982+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7eff04016070 con 0x7eff0c0c9150 2023-11-08T08:21:57.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.982+0000 7eff10c2d700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efef00682f0 0x7efef006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:57.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.982+0000 7eff10c2d700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efef00682f0 0x7efef006a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0c8990 tx=0x7eff00010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:57.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:57.986+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff04055620 con 0x7eff0c0c9150 2023-11-08T08:21:58.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.164+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7eff0c0743f0 con 0x7efef00682f0 2023-11-08T08:21:58.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.165+0000 7eff097fa700 1 -- 172.21.15.116:0/4054867147 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7eff0c0743f0 con 0x7efef00682f0 2023-11-08T08:21:58.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efef00682f0 msgr2=0x7efef006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efef00682f0 0x7efef006a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0c8990 tx=0x7eff00010000 comp rx=0 tx=0).stop 2023-11-08T08:21:58.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 msgr2=0x7eff0c0c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 0x7eff0c0c7b80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7eff04009fb0 tx=0x7eff0400e5b0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 shutdown_connections 2023-11-08T08:21:58.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c0c9150 0x7eff0c0c7b80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7efef00682f0 0x7efef006a7a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff0c0af0a0 0x7eff0c14e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 --2- 172.21.15.116:0/4054867147 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7eff0c0b1650 0x7eff0c0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 >> 172.21.15.116:0/4054867147 conn(0x7eff0c06c010 msgr2=0x7eff0c159140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 shutdown_connections 2023-11-08T08:21:58.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.167+0000 7eff12e91700 1 -- 172.21.15.116:0/4054867147 wait complete. 2023-11-08T08:21:58.172 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 2 to deep-scrub 2023-11-08T08:21:58.180 INFO:tasks.ceph:Scrubbing osd.3 2023-11-08T08:21:58.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:58.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.302+0000 7f1b8d896700 1 Processor -- start 2023-11-08T08:21:58.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.302+0000 7f1b8d896700 1 -- start start 2023-11-08T08:21:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b8d896700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1b880ceea0 0x7f1b8815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b8d896700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b880cea10 con 0x7f1b8815e070 2023-11-08T08:21:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b8d896700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b880cbd10 con 0x7f1b8806ba90 2023-11-08T08:21:58.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b8d896700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8815ae80 con 0x7f1b880ceea0 2023-11-08T08:21:58.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b86ffd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1b8815e070 0x7f1b8806b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58592/0 (socket says 172.21.15.116:58592) 2023-11-08T08:21:58.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b86ffd700 1 -- 172.21.15.116:0/305739425 learned_addr learned my addr 172.21.15.116:0/305739425 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:58.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b877fe700 1 --2- 172.21.15.116:0/305739425 >> v2:172.21.15.129:3300/0 conn(0x7f1b880ceea0 0x7f1b8815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216388457 0 0) 0x7f1b880cea10 con 0x7f1b8815e070 2023-11-08T08:21:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.303+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b700035b0 con 0x7f1b8815e070 2023-11-08T08:21:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467791045 0 0) 0x7f1b700035b0 con 0x7f1b8815e070 2023-11-08T08:21:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 >> v2:172.21.15.129:3300/0 conn(0x7f1b880ceea0 msgr2=0x7f1b8815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 --2- 172.21.15.116:0/305739425 >> v2:172.21.15.129:3300/0 conn(0x7f1b880ceea0 0x7f1b8815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 >> v1:172.21.15.129:6789/0 conn(0x7f1b8806ba90 legacy=0x7f1b8815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880ce0e0 con 0x7f1b8815e070 2023-11-08T08:21:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f1b7c001670 con 0x7f1b8815e070 2023-11-08T08:21:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 >> v1:172.21.15.116:6789/0 conn(0x7f1b8815e070 legacy=0x7f1b8806b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b70003a00 0x7f1b70003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b70004850 0x7f1b70004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70005200 0x7f1b7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8815ae80 con 0x7f1b70003a00 2023-11-08T08:21:58.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7000c080 con 0x7f1b70005200 2023-11-08T08:21:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7000c240 con 0x7f1b70004850 2023-11-08T08:21:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b70003a00 0x7f1b70003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b877fe700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70005200 0x7f1b7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b867fc700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b70004850 0x7f1b70004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 -- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b70004850 msgr2=0x7f1b70004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b70004850 0x7f1b70004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 -- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70005200 msgr2=0x7f1b7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70005200 0x7f1b7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b877fe700 1 -- 172.21.15.116:0/305739425 reap_dead start 2023-11-08T08:21:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 -- 172.21.15.116:0/305739425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700035b0 con 0x7f1b70003a00 2023-11-08T08:21:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b86ffd700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b70003a00 0x7f1b70003e10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00c240 tx=0x7f1b70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1663e77d9908f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c007c10 con 0x7f1b70003a00 2023-11-08T08:21:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c00cd60 con 0x7f1b70003a00 2023-11-08T08:21:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.304+0000 7f1b85ffb700 1 -- 172.21.15.116:0/305739425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c005ba0 con 0x7f1b70003a00 2023-11-08T08:21:58.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 -- 172.21.15.116:0/305739425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b70003a00 msgr2=0x7f1b70003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b70003a00 0x7f1b70003e10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00c240 tx=0x7f1b70005880 comp rx=0 tx=0).stop 2023-11-08T08:21:58.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 -- 172.21.15.116:0/305739425 shutdown_connections 2023-11-08T08:21:58.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 --2- 172.21.15.116:0/305739425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b70003a00 0x7f1b70003e10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 -- 172.21.15.116:0/305739425 >> 172.21.15.116:0/305739425 conn(0x7f1b880cfc30 msgr2=0x7f1b880d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 -- 172.21.15.116:0/305739425 shutdown_connections 2023-11-08T08:21:58.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.305+0000 7f1b8d896700 1 -- 172.21.15.116:0/305739425 wait complete. 2023-11-08T08:21:58.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.306+0000 7f1b8d896700 1 Processor -- start 2023-11-08T08:21:58.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.306+0000 7f1b8d896700 1 -- start start 2023-11-08T08:21:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b8d896700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70003a00 0x7f1b881400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b8d896700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b88150200 0x7f1b8806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b8d896700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b88143fc0 0x7f1b88144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b8d896700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8806ef60 con 0x7f1b88143fc0 2023-11-08T08:21:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b8d896700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88144970 con 0x7f1b70003a00 2023-11-08T08:21:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b8d896700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88140610 con 0x7f1b88150200 2023-11-08T08:21:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b88143fc0 0x7f1b88144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b88143fc0 0x7f1b88144430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53964/0 (socket says 172.21.15.116:53964) 2023-11-08T08:21:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 -- 172.21.15.116:0/2006611520 learned_addr learned my addr 172.21.15.116:0/2006611520 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 -- 172.21.15.116:0/2006611520 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b88150200 msgr2=0x7f1b8806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1b88150200 0x7f1b8806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 -- 172.21.15.116:0/2006611520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70003a00 msgr2=0x7f1b881400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b70003a00 0x7f1b881400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.307+0000 7f1b877fe700 1 -- 172.21.15.116:0/2006611520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880ce6a0 con 0x7f1b88143fc0 2023-11-08T08:21:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b877fe700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b88143fc0 0x7f1b88144430 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1b80011bf0 tx=0x7f1b8000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000e040 con 0x7f1b88143fc0 2023-11-08T08:21:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8000f040 con 0x7f1b88143fc0 2023-11-08T08:21:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b880ce3c0 con 0x7f1b88143fc0 2023-11-08T08:21:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000fdf0 con 0x7f1b88143fc0 2023-11-08T08:21:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8815a150 con 0x7f1b88143fc0 2023-11-08T08:21:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.308+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1b880743f0 con 0x7f1b88143fc0 2023-11-08T08:21:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.309+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b80022460 con 0x7f1b88143fc0 2023-11-08T08:21:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.309+0000 7f1b6ffff700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.309+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f1b80086d00 con 0x7f1b88143fc0 2023-11-08T08:21:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.309+0000 7f1b86ffd700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.310+0000 7f1b6ffff700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7f1b7406e760 0x7f1b74070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.310+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b74071240 con 0x7f1b7406e760 2023-11-08T08:21:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.310+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b80087140 con 0x7f1b88143fc0 2023-11-08T08:21:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.310+0000 7f1b867fc700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7f1b7406e760 0x7f1b74070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.310+0000 7f1b86ffd700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1b740682f0 0x7f1b7406a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c000c00 tx=0x7f1b7c00a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.310+0000 7f1b867fc700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7f1b7406e760 0x7f1b74070b70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.311+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== osd.3 v2:172.21.15.116:6808/121155 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1b74071240 con 0x7f1b7406e760 2023-11-08T08:21:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.324+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 --> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1b880c5310 con 0x7f1b7406e760 2023-11-08T08:21:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b6ffff700 1 -- 172.21.15.116:0/2006611520 <== osd.3 v2:172.21.15.116:6808/121155 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1b880c5310 con 0x7f1b7406e760 2023-11-08T08:21:58.334 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:58.334 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:58.334 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7f1b7406e760 msgr2=0x7f1b74070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6808/121155,v1:172.21.15.116:6809/121155] conn(0x7f1b7406e760 0x7f1b74070b70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1b740682f0 msgr2=0x7f1b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f1b740682f0 0x7f1b7406a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c000c00 tx=0x7f1b7c00a000 comp rx=0 tx=0).stop 2023-11-08T08:21:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b88143fc0 msgr2=0x7f1b88144430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 --2- 172.21.15.116:0/2006611520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b88143fc0 0x7f1b88144430 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1b80011bf0 tx=0x7f1b8000f1b0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b877fe700 1 -- 172.21.15.116:0/2006611520 reap_dead start 2023-11-08T08:21:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 shutdown_connections 2023-11-08T08:21:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 >> 172.21.15.116:0/2006611520 conn(0x7f1b880cfc30 msgr2=0x7f1b880d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 shutdown_connections 2023-11-08T08:21:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.325+0000 7f1b8d896700 1 -- 172.21.15.116:0/2006611520 wait complete. 2023-11-08T08:21:58.340 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-08T08:21:58.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.459+0000 7f18a96e3700 1 Processor -- start 2023-11-08T08:21:58.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.459+0000 7f18a96e3700 1 -- start start 2023-11-08T08:21:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a96e3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a40cf710 0x7f18a40ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a96e3700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18a406a980 0x7f18a4071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a96e3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a415cc10 con 0x7f18a4071890 2023-11-08T08:21:58.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a96e3700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a406c9b0 con 0x7f18a406a980 2023-11-08T08:21:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a96e3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a406cc90 con 0x7f18a40cf710 2023-11-08T08:21:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a37fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18a406a980 0x7f18a4071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a27fc700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f18a4071890 0x7f18a406a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58604/0 (socket says 172.21.15.116:58604) 2023-11-08T08:21:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a27fc700 1 -- 172.21.15.116:0/913732291 learned_addr learned my addr 172.21.15.116:0/913732291 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a37fe700 1 -- 172.21.15.116:0/913732291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a40cf710 msgr2=0x7f18a40ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.460+0000 7f18a2ffd700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a40cf710 0x7f18a40ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a37fe700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a40cf710 0x7f18a40ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a1ffb700 1 -- 172.21.15.116:0/913732291 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449235525 0 0) 0x7f18a415cc10 con 0x7f18a4071890 2023-11-08T08:21:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a37fe700 1 -- 172.21.15.116:0/913732291 >> v1:172.21.15.116:6789/0 conn(0x7f18a4071890 legacy=0x7f18a406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a37fe700 1 -- 172.21.15.116:0/913732291 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a406cf70 con 0x7f18a406a980 2023-11-08T08:21:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a2ffd700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a40cf710 0x7f18a40ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a37fe700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a406a980 0x7f18a4071220 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f189c009d90 tx=0x7f189c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c36ec2fc2bf41a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a1ffb700 1 -- 172.21.15.116:0/913732291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c00e070 con 0x7f18a406a980 2023-11-08T08:21:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a1ffb700 1 -- 172.21.15.116:0/913732291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c00bbd0 con 0x7f18a406a980 2023-11-08T08:21:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a1ffb700 1 -- 172.21.15.116:0/913732291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c012430 con 0x7f18a406a980 2023-11-08T08:21:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 -- 172.21.15.116:0/913732291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a406a980 msgr2=0x7f18a4071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a406a980 0x7f18a4071220 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f189c009d90 tx=0x7f189c00b330 comp rx=0 tx=0).stop 2023-11-08T08:21:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 -- 172.21.15.116:0/913732291 shutdown_connections 2023-11-08T08:21:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a406a980 0x7f18a4071220 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 --2- 172.21.15.116:0/913732291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a40cf710 0x7f18a40ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 -- 172.21.15.116:0/913732291 >> 172.21.15.116:0/913732291 conn(0x7f18a406c010 msgr2=0x7f18a40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 -- 172.21.15.116:0/913732291 shutdown_connections 2023-11-08T08:21:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.461+0000 7f18a96e3700 1 -- 172.21.15.116:0/913732291 wait complete. 2023-11-08T08:21:58.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 Processor -- start 2023-11-08T08:21:58.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 -- start start 2023-11-08T08:21:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a406a980 0x7f18a414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a4071890 0x7f18a40c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 0x7f18a40afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a415da70 con 0x7f18a40cf710 2023-11-08T08:21:58.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a414e500 con 0x7f18a4071890 2023-11-08T08:21:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.463+0000 7f18a96e3700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a415d440 con 0x7f18a406a980 2023-11-08T08:21:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 0x7f18a40afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 0x7f18a40afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53980/0 (socket says 172.21.15.116:53980) 2023-11-08T08:21:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 -- 172.21.15.116:0/1304443700 learned_addr learned my addr 172.21.15.116:0/1304443700 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a27fc700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a4071890 0x7f18a40c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a2ffd700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a406a980 0x7f18a414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 -- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a406a980 msgr2=0x7f18a414dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a406a980 0x7f18a414dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 -- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a4071890 msgr2=0x7f18a40c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a4071890 0x7f18a40c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 -- 172.21.15.116:0/1304443700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c009a70 con 0x7f18a40cf710 2023-11-08T08:21:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a27fc700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a4071890 0x7f18a40c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a2ffd700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a406a980 0x7f18a414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a37fe700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 0x7f18a40afd20 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f189c013000 tx=0x7f189c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c003ea0 con 0x7f18a40cf710 2023-11-08T08:21:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c020070 con 0x7f18a40cf710 2023-11-08T08:21:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a406d530 con 0x7f18a40cf710 2023-11-08T08:21:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c004280 con 0x7f18a40cf710 2023-11-08T08:21:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.464+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a406d250 con 0x7f18a40cf710 2023-11-08T08:21:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.465+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a4160480 con 0x7f18a40cf710 2023-11-08T08:21:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.466+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f189c0043e0 con 0x7f18a40cf710 2023-11-08T08:21:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.466+0000 7f188b7fe700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18900682f0 0x7f189006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.466+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f189c022070 con 0x7f18a40cf710 2023-11-08T08:21:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.466+0000 7f18a2ffd700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18900682f0 0x7f189006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.466+0000 7f18a2ffd700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18900682f0 0x7f189006a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f189800aa20 tx=0x7f1898008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.470+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f189c054a80 con 0x7f18a40cf710 2023-11-08T08:21:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.651+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f18a40b1720 con 0x7f18900682f0 2023-11-08T08:21:58.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.651+0000 7f188b7fe700 1 -- 172.21.15.116:0/1304443700 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f18a40b1720 con 0x7f18900682f0 2023-11-08T08:21:58.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18900682f0 msgr2=0x7f189006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18900682f0 0x7f189006a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f189800aa20 tx=0x7f1898008040 comp rx=0 tx=0).stop 2023-11-08T08:21:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 msgr2=0x7f18a40afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 0x7f18a40afd20 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f189c013000 tx=0x7f189c004ad0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 shutdown_connections 2023-11-08T08:21:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a40cf710 0x7f18a40afd20 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18a406a980 0x7f18a414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18900682f0 0x7f189006a7a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 --2- 172.21.15.116:0/1304443700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a4071890 0x7f18a40c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 >> 172.21.15.116:0/1304443700 conn(0x7f18a406c010 msgr2=0x7f18a41e55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 shutdown_connections 2023-11-08T08:21:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.654+0000 7f18a96e3700 1 -- 172.21.15.116:0/1304443700 wait complete. 2023-11-08T08:21:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 3 to deep-scrub 2023-11-08T08:21:58.667 INFO:tasks.ceph:Scrubbing osd.4 2023-11-08T08:21:58.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:58.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.790+0000 7f76c63a5700 1 Processor -- start 2023-11-08T08:21:58.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76c63a5700 1 -- start start 2023-11-08T08:21:58.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76c63a5700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76c00cb4b0 0x7f76c0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76c63a5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c006c390 0x7f76c006c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76c63a5700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c006cce0 con 0x7f76c00ce7d0 2023-11-08T08:21:58.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76c63a5700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c00ca0e0 con 0x7f76c00cb4b0 2023-11-08T08:21:58.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76c63a5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c006a270 con 0x7f76c006c390 2023-11-08T08:21:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76bf7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76c00cb4b0 0x7f76c0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.791+0000 7f76bffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f76c00ce7d0 0x7f76c0070380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58618/0 (socket says 172.21.15.116:58618) 2023-11-08T08:21:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76bffff700 1 -- 172.21.15.116:0/1513423866 learned_addr learned my addr 172.21.15.116:0/1513423866 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76c4942700 1 --2- 172.21.15.116:0/1513423866 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c006c390 0x7f76c006c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76bf7fe700 1 -- 172.21.15.116:0/1513423866 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c006c390 msgr2=0x7f76c006c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76bf7fe700 1 --2- 172.21.15.116:0/1513423866 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c006c390 0x7f76c006c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76bf7fe700 1 -- 172.21.15.116:0/1513423866 >> v1:172.21.15.116:6789/0 conn(0x7f76c00ce7d0 legacy=0x7f76c0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76bf7fe700 1 -- 172.21.15.116:0/1513423866 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c00ceba0 con 0x7f76c00cb4b0 2023-11-08T08:21:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76bf7fe700 1 --2- 172.21.15.116:0/1513423866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cb4b0 0x7f76c0159ab0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f76b400bae0 tx=0x7f76b400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cee5d08168dc50 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76beffd700 1 -- 172.21.15.116:0/1513423866 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b400dc20 con 0x7f76c00cb4b0 2023-11-08T08:21:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76beffd700 1 -- 172.21.15.116:0/1513423866 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b400dd80 con 0x7f76c00cb4b0 2023-11-08T08:21:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.792+0000 7f76beffd700 1 -- 172.21.15.116:0/1513423866 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4017630 con 0x7f76c00cb4b0 2023-11-08T08:21:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 -- 172.21.15.116:0/1513423866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cb4b0 msgr2=0x7f76c0159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 --2- 172.21.15.116:0/1513423866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cb4b0 0x7f76c0159ab0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f76b400bae0 tx=0x7f76b400bdf0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 -- 172.21.15.116:0/1513423866 shutdown_connections 2023-11-08T08:21:58.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 --2- 172.21.15.116:0/1513423866 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c006c390 0x7f76c006c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 --2- 172.21.15.116:0/1513423866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cb4b0 0x7f76c0159ab0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 -- 172.21.15.116:0/1513423866 >> 172.21.15.116:0/1513423866 conn(0x7f76c00cf370 msgr2=0x7f76c00cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 -- 172.21.15.116:0/1513423866 shutdown_connections 2023-11-08T08:21:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.793+0000 7f76c63a5700 1 -- 172.21.15.116:0/1513423866 wait complete. 2023-11-08T08:21:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.794+0000 7f76c63a5700 1 Processor -- start 2023-11-08T08:21:58.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.794+0000 7f76c63a5700 1 -- start start 2023-11-08T08:21:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76c006c390 0x7f76c00c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c00cb4b0 0x7f76c01579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00ce7d0 0x7f76c0157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00c5bb0 con 0x7f76c00ce7d0 2023-11-08T08:21:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00c5210 con 0x7f76c006c390 2023-11-08T08:21:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00c54f0 con 0x7f76c00cb4b0 2023-11-08T08:21:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00ce7d0 0x7f76c0157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00ce7d0 0x7f76c0157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54002/0 (socket says 172.21.15.116:54002) 2023-11-08T08:21:58.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 -- 172.21.15.116:0/806105560 learned_addr learned my addr 172.21.15.116:0/806105560 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:58.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bf7fe700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c00cb4b0 0x7f76c01579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bffff700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76c006c390 0x7f76c00c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 -- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c00cb4b0 msgr2=0x7f76c01579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c00cb4b0 0x7f76c01579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 -- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76c006c390 msgr2=0x7f76c00c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76c006c390 0x7f76c00c4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 -- 172.21.15.116:0/806105560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b400b740 con 0x7f76c00ce7d0 2023-11-08T08:21:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bffff700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76c006c390 0x7f76c00c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bf7fe700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76c00cb4b0 0x7f76c01579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c4942700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00ce7d0 0x7f76c0157f10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f76b8007e40 tx=0x7f76b800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b800eea0 con 0x7f76c00ce7d0 2023-11-08T08:21:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8014420 con 0x7f76c00ce7d0 2023-11-08T08:21:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c00cb950 con 0x7f76c00ce7d0 2023-11-08T08:21:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80147b0 con 0x7f76c00ce7d0 2023-11-08T08:21:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.795+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c00c9b40 con 0x7f76c00ce7d0 2023-11-08T08:21:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.796+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76c00c9e00 con 0x7f76c00ce7d0 2023-11-08T08:21:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f76b8012050 con 0x7f76c00ce7d0 2023-11-08T08:21:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bcff9700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f76ac0682f0 0x7f76ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f76b8086060 con 0x7f76c00ce7d0 2023-11-08T08:21:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bffff700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f76ac0682f0 0x7f76ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bcff9700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f76ac06e760 0x7f76ac070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 --> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76ac071240 con 0x7f76ac06e760 2023-11-08T08:21:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76b8088710 con 0x7f76c00ce7d0 2023-11-08T08:21:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.797+0000 7f76bffff700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f76ac0682f0 0x7f76ac06a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f76b0005910 tx=0x7f76b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.798+0000 7f76bf7fe700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f76ac06e760 0x7f76ac070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.798+0000 7f76bf7fe700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f76ac06e760 0x7f76ac070b70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.799+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== osd.4 v2:172.21.15.129:6800/121420 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f76ac071240 con 0x7f76ac06e760 2023-11-08T08:21:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 --> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f76c00c5cf0 con 0x7f76ac06e760 2023-11-08T08:21:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76bcff9700 1 -- 172.21.15.116:0/806105560 <== osd.4 v2:172.21.15.129:6800/121420 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f76c00c5cf0 con 0x7f76ac06e760 2023-11-08T08:21:58.820 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:58.820 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:58.820 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f76ac06e760 msgr2=0x7f76ac070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.129:6800/121420,v1:172.21.15.129:6801/121420] conn(0x7f76ac06e760 0x7f76ac070b70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f76ac0682f0 msgr2=0x7f76ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f76ac0682f0 0x7f76ac06a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f76b0005910 tx=0x7f76b0009040 comp rx=0 tx=0).stop 2023-11-08T08:21:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00ce7d0 msgr2=0x7f76c0157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 --2- 172.21.15.116:0/806105560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00ce7d0 0x7f76c0157f10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f76b8007e40 tx=0x7f76b800e3c0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c4942700 1 -- 172.21.15.116:0/806105560 reap_dead start 2023-11-08T08:21:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 shutdown_connections 2023-11-08T08:21:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.813+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 >> 172.21.15.116:0/806105560 conn(0x7f76c00cf370 msgr2=0x7f76c015b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.814+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 shutdown_connections 2023-11-08T08:21:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.814+0000 7f76c63a5700 1 -- 172.21.15.116:0/806105560 wait complete. 2023-11-08T08:21:58.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-08T08:21:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.946+0000 7f18b7549700 1 Processor -- start 2023-11-08T08:21:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.946+0000 7f18b7549700 1 -- start start 2023-11-08T08:21:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b7549700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18b00cf710 0x7f18b00ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b7549700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b006a980 0x7f18b0071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b7549700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b015cc10 con 0x7f18b0071890 2023-11-08T08:21:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b7549700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b006c9b0 con 0x7f18b00cf710 2023-11-08T08:21:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b7549700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b006cc90 con 0x7f18b006a980 2023-11-08T08:21:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b52e5700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18b00cf710 0x7f18b00ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b4ae4700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f18b0071890 0x7f18b006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58626/0 (socket says 172.21.15.116:58626) 2023-11-08T08:21:58.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b4ae4700 1 -- 172.21.15.116:0/4160550417 learned_addr learned my addr 172.21.15.116:0/4160550417 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:58.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.947+0000 7f18b5ae6700 1 --2- 172.21.15.116:0/4160550417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b006a980 0x7f18b0071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b52e5700 1 -- 172.21.15.116:0/4160550417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b006a980 msgr2=0x7f18b0071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b52e5700 1 --2- 172.21.15.116:0/4160550417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b006a980 0x7f18b0071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18a7fff700 1 -- 172.21.15.116:0/4160550417 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790329106 0 0) 0x7f18b015cc10 con 0x7f18b0071890 2023-11-08T08:21:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b52e5700 1 -- 172.21.15.116:0/4160550417 >> v1:172.21.15.116:6789/0 conn(0x7f18b0071890 legacy=0x7f18b006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b52e5700 1 -- 172.21.15.116:0/4160550417 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b006cf70 con 0x7f18b00cf710 2023-11-08T08:21:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b52e5700 1 --2- 172.21.15.116:0/4160550417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b00cf710 0x7f18b00ce740 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009870 tx=0x7f18a0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc0a09dbfe50c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18a7fff700 1 -- 172.21.15.116:0/4160550417 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0011830 con 0x7f18b00cf710 2023-11-08T08:21:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18a7fff700 1 -- 172.21.15.116:0/4160550417 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a0011990 con 0x7f18b00cf710 2023-11-08T08:21:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18a7fff700 1 -- 172.21.15.116:0/4160550417 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0011d20 con 0x7f18b00cf710 2023-11-08T08:21:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 -- 172.21.15.116:0/4160550417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b00cf710 msgr2=0x7f18b00ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 --2- 172.21.15.116:0/4160550417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b00cf710 0x7f18b00ce740 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009870 tx=0x7f18a0009b80 comp rx=0 tx=0).stop 2023-11-08T08:21:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 -- 172.21.15.116:0/4160550417 shutdown_connections 2023-11-08T08:21:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 --2- 172.21.15.116:0/4160550417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b00cf710 0x7f18b00ce740 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 --2- 172.21.15.116:0/4160550417 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b006a980 0x7f18b0071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 -- 172.21.15.116:0/4160550417 >> 172.21.15.116:0/4160550417 conn(0x7f18b006c010 msgr2=0x7f18b00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 -- 172.21.15.116:0/4160550417 shutdown_connections 2023-11-08T08:21:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.948+0000 7f18b7549700 1 -- 172.21.15.116:0/4160550417 wait complete. 2023-11-08T08:21:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 Processor -- start 2023-11-08T08:21:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 -- start start 2023-11-08T08:21:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 0x7f18b00b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b0071890 0x7f18b015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b00cf710 0x7f18b0147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b0148b60 con 0x7f18b006a980 2023-11-08T08:21:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b0148e40 con 0x7f18b0071890 2023-11-08T08:21:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b7549700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b0149120 con 0x7f18b00cf710 2023-11-08T08:21:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b52e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 0x7f18b00b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b52e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 0x7f18b00b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54028/0 (socket says 172.21.15.116:54028) 2023-11-08T08:21:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.950+0000 7f18b52e5700 1 -- 172.21.15.116:0/646655167 learned_addr learned my addr 172.21.15.116:0/646655167 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b52e5700 1 -- 172.21.15.116:0/646655167 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b00cf710 msgr2=0x7f18b0147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b52e5700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b00cf710 0x7f18b0147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b52e5700 1 -- 172.21.15.116:0/646655167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b0071890 msgr2=0x7f18b015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b52e5700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b0071890 0x7f18b015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b52e5700 1 -- 172.21.15.116:0/646655167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00094a0 con 0x7f18b006a980 2023-11-08T08:21:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b52e5700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 0x7f18b00b4350 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f18a0014040 tx=0x7f18a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a00036a0 con 0x7f18b006a980 2023-11-08T08:21:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a001e070 con 0x7f18b006a980 2023-11-08T08:21:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b006d530 con 0x7f18b006a980 2023-11-08T08:21:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0003a20 con 0x7f18b006a980 2023-11-08T08:21:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.951+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b006d250 con 0x7f18b006a980 2023-11-08T08:21:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.952+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b0160480 con 0x7f18b006a980 2023-11-08T08:21:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.952+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18a000f050 con 0x7f18b006a980 2023-11-08T08:21:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.953+0000 7f18a5ffb700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18940682f0 0x7f189406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.953+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f18a00869b0 con 0x7f18b006a980 2023-11-08T08:21:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.953+0000 7f18b4ae4700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18940682f0 0x7f189406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.953+0000 7f18b4ae4700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18940682f0 0x7f189406a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0011e0 tx=0x7f18ac0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:58.957+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18a0053510 con 0x7f18b006a980 2023-11-08T08:21:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.137+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b00b0d50 con 0x7f18940682f0 2023-11-08T08:21:59.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.138+0000 7f18a5ffb700 1 -- 172.21.15.116:0/646655167 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f18b00b0d50 con 0x7f18940682f0 2023-11-08T08:21:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18940682f0 msgr2=0x7f189406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18940682f0 0x7f189406a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0011e0 tx=0x7f18ac0079b0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 msgr2=0x7f18b00b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 0x7f18b00b4350 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f18a0014040 tx=0x7f18a0004750 comp rx=0 tx=0).stop 2023-11-08T08:21:59.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 shutdown_connections 2023-11-08T08:21:59.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f18b00cf710 0x7f18b0147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b006a980 0x7f18b00b4350 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f18940682f0 0x7f189406a7a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 --2- 172.21.15.116:0/646655167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b0071890 0x7f18b015ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 >> 172.21.15.116:0/646655167 conn(0x7f18b006c010 msgr2=0x7f18b00ac8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 shutdown_connections 2023-11-08T08:21:59.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.140+0000 7f18b7549700 1 -- 172.21.15.116:0/646655167 wait complete. 2023-11-08T08:21:59.145 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 4 to deep-scrub 2023-11-08T08:21:59.152 INFO:tasks.ceph:Scrubbing osd.5 2023-11-08T08:21:59.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.275+0000 7f5626429700 1 Processor -- start 2023-11-08T08:21:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.275+0000 7f5626429700 1 -- start start 2023-11-08T08:21:59.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f5626429700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f562006ba90 0x7f562015a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f5626429700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200cea10 con 0x7f562015e070 2023-11-08T08:21:59.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f5626429700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200cbd10 con 0x7f56200ceea0 2023-11-08T08:21:59.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f5626429700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562015ae80 con 0x7f562006ba90 2023-11-08T08:21:59.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f561ffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f562015e070 0x7f562006b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58642/0 (socket says 172.21.15.116:58642) 2023-11-08T08:21:59.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f561ffff700 1 -- 172.21.15.116:0/3035941980 learned_addr learned my addr 172.21.15.116:0/3035941980 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:59.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3035941980 >> v2:172.21.15.129:3300/0 conn(0x7f562006ba90 0x7f562015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765570215 0 0) 0x7f56200cea10 con 0x7f562015e070 2023-11-08T08:21:59.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.276+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56080035b0 con 0x7f562015e070 2023-11-08T08:21:59.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066707743 0 0) 0x7f56080035b0 con 0x7f562015e070 2023-11-08T08:21:59.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 >> v2:172.21.15.129:3300/0 conn(0x7f562006ba90 msgr2=0x7f562015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:59.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 --2- 172.21.15.116:0/3035941980 >> v2:172.21.15.129:3300/0 conn(0x7f562006ba90 0x7f562015a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 >> v1:172.21.15.129:6789/0 conn(0x7f56200ceea0 legacy=0x7f562015a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200ce0e0 con 0x7f562015e070 2023-11-08T08:21:59.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1901139781 0 0) 0x7f5614001670 con 0x7f562015e070 2023-11-08T08:21:59.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 >> v1:172.21.15.116:6789/0 conn(0x7f562015e070 legacy=0x7f562006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5608003a50 0x7f5608003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f5608004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5608005200 0x7f560800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562015ae80 con 0x7f5608004850 2023-11-08T08:21:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560800c0e0 con 0x7f5608003a50 2023-11-08T08:21:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560800c2a0 con 0x7f5608005200 2023-11-08T08:21:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f5608004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561ffff700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5608003a50 0x7f5608003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f56249c6700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5608005200 0x7f560800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 -- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5608005200 msgr2=0x7f560800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5608005200 0x7f560800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 -- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5608003a50 msgr2=0x7f5608003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5608003a50 0x7f5608003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 -- 172.21.15.116:0/3035941980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56080035b0 con 0x7f5608004850 2023-11-08T08:21:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561ffff700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5608003a50 0x7f5608003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f56249c6700 1 -- 172.21.15.116:0/3035941980 reap_dead start 2023-11-08T08:21:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f5608004cc0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f562006be80 tx=0x7f5608005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f3362d2f0974c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.277+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561000e6c0 con 0x7f5608004850 2023-11-08T08:21:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561000e820 con 0x7f5608004850 2023-11-08T08:21:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f561effd700 1 -- 172.21.15.116:0/3035941980 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561000ebb0 con 0x7f5608004850 2023-11-08T08:21:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 -- 172.21.15.116:0/3035941980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 msgr2=0x7f5608004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f5608004cc0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f562006be80 tx=0x7f5608005880 comp rx=0 tx=0).stop 2023-11-08T08:21:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 -- 172.21.15.116:0/3035941980 shutdown_connections 2023-11-08T08:21:59.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 --2- 172.21.15.116:0/3035941980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f5608004cc0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 -- 172.21.15.116:0/3035941980 >> 172.21.15.116:0/3035941980 conn(0x7f56200cfc30 msgr2=0x7f56200d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 -- 172.21.15.116:0/3035941980 shutdown_connections 2023-11-08T08:21:59.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.278+0000 7f5626429700 1 -- 172.21.15.116:0/3035941980 wait complete. 2023-11-08T08:21:59.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.279+0000 7f5626429700 1 Processor -- start 2023-11-08T08:21:59.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 -- start start 2023-11-08T08:21:59.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f562014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f56200ad2f0 0x7f562006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f562006dde0 0x7f562006a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200ad760 con 0x7f5608004850 2023-11-08T08:21:59.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200ada40 con 0x7f562006dde0 2023-11-08T08:21:59.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f5626429700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56200add20 con 0x7f56200ad2f0 2023-11-08T08:21:59.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f561ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f562014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f561ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f562014e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54060/0 (socket says 172.21.15.116:54060) 2023-11-08T08:21:59.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.280+0000 7f561ffff700 1 -- 172.21.15.116:0/3595080622 learned_addr learned my addr 172.21.15.116:0/3595080622 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:59.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561ffff700 1 -- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f56200ad2f0 msgr2=0x7f562006d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561ffff700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f56200ad2f0 0x7f562006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561ffff700 1 -- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f562006dde0 msgr2=0x7f562006a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561ffff700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f562006dde0 0x7f562006a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561ffff700 1 -- 172.21.15.116:0/3595080622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200ce6a0 con 0x7f5608004850 2023-11-08T08:21:59.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561ffff700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f562014e6b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5614000fb0 tx=0x7f561400eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614012590 con 0x7f5608004850 2023-11-08T08:21:59.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56140126f0 con 0x7f5608004850 2023-11-08T08:21:59.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56200ce3c0 con 0x7f5608004850 2023-11-08T08:21:59.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614012a60 con 0x7f5608004850 2023-11-08T08:21:59.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.281+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562006b380 con 0x7f5608004850 2023-11-08T08:21:59.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.282+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f56200743f0 con 0x7f5608004850 2023-11-08T08:21:59.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.282+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5614012bc0 con 0x7f5608004850 2023-11-08T08:21:59.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.282+0000 7f561cff9700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f560c0682f0 0x7f560c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f56140858c0 con 0x7f5608004850 2023-11-08T08:21:59.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f560c0682f0 0x7f560c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f561cff9700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f560c06e760 0x7f560c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 --> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c071240 con 0x7f560c06e760 2023-11-08T08:21:59.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f561408a2a0 con 0x7f5608004850 2023-11-08T08:21:59.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f56249c6700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f560c06e760 0x7f560c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f561f7fe700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f560c0682f0 0x7f560c06a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f562006e420 tx=0x7f5610013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.283+0000 7f56249c6700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f560c06e760 0x7f560c070b70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.284+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== osd.5 v2:172.21.15.129:6808/121421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f560c071240 con 0x7f560c06e760 2023-11-08T08:21:59.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.298+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 --> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f56200b30f0 con 0x7f560c06e760 2023-11-08T08:21:59.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.298+0000 7f561cff9700 1 -- 172.21.15.116:0/3595080622 <== osd.5 v2:172.21.15.129:6808/121421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f56200b30f0 con 0x7f560c06e760 2023-11-08T08:21:59.308 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:59.309 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.298+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f560c06e760 msgr2=0x7f560c070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.298+0000 7f5626429700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.129:6808/121421,v1:172.21.15.129:6809/121421] conn(0x7f560c06e760 0x7f560c070b70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f560c0682f0 msgr2=0x7f560c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f560c0682f0 0x7f560c06a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f562006e420 tx=0x7f5610013000 comp rx=0 tx=0).stop 2023-11-08T08:21:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 msgr2=0x7f562014e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 --2- 172.21.15.116:0/3595080622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5608004850 0x7f562014e6b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5614000fb0 tx=0x7f561400eb10 comp rx=0 tx=0).stop 2023-11-08T08:21:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f56249c6700 1 -- 172.21.15.116:0/3595080622 reap_dead start 2023-11-08T08:21:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 shutdown_connections 2023-11-08T08:21:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 >> 172.21.15.116:0/3595080622 conn(0x7f56200cfc30 msgr2=0x7f56200c5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 shutdown_connections 2023-11-08T08:21:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.299+0000 7f5626429700 1 -- 172.21.15.116:0/3595080622 wait complete. 2023-11-08T08:21:59.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-08T08:21:59.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 Processor -- start 2023-11-08T08:21:59.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 -- start start 2023-11-08T08:21:59.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1300b0540 0x7fb1300c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300ca310 0x7fb1300b0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13015a8c0 con 0x7fb13015e910 2023-11-08T08:21:59.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1300ca780 con 0x7fb1300b0540 2023-11-08T08:21:59.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.434+0000 7fb135403700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300b0e50 con 0x7fb1300ca310 2023-11-08T08:21:59.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12effd700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1300b0540 0x7fb1300c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12f7fe700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb13015e910 0x7fb130070940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58646/0 (socket says 172.21.15.116:58646) 2023-11-08T08:21:59.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12f7fe700 1 -- 172.21.15.116:0/570244470 learned_addr learned my addr 172.21.15.116:0/570244470 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12e7fc700 1 --2- 172.21.15.116:0/570244470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300ca310 0x7fb1300b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12dffb700 1 -- 172.21.15.116:0/570244470 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038675375 0 0) 0x7fb13015a8c0 con 0x7fb13015e910 2023-11-08T08:21:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12effd700 1 -- 172.21.15.116:0/570244470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300ca310 msgr2=0x7fb1300b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12effd700 1 --2- 172.21.15.116:0/570244470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300ca310 0x7fb1300b0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12effd700 1 -- 172.21.15.116:0/570244470 >> v1:172.21.15.116:6789/0 conn(0x7fb13015e910 legacy=0x7fb130070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12effd700 1 -- 172.21.15.116:0/570244470 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300caa10 con 0x7fb1300b0540 2023-11-08T08:21:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12effd700 1 --2- 172.21.15.116:0/570244470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300b0540 0x7fb1300c9bf0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb120009840 tx=0x7fb120009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1ecf63050a595c server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12dffb700 1 -- 172.21.15.116:0/570244470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120011830 con 0x7fb1300b0540 2023-11-08T08:21:59.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12dffb700 1 -- 172.21.15.116:0/570244470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb120011990 con 0x7fb1300b0540 2023-11-08T08:21:59.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.435+0000 7fb12dffb700 1 -- 172.21.15.116:0/570244470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120011d20 con 0x7fb1300b0540 2023-11-08T08:21:59.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 -- 172.21.15.116:0/570244470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300b0540 msgr2=0x7fb1300c9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 --2- 172.21.15.116:0/570244470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300b0540 0x7fb1300c9bf0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb120009840 tx=0x7fb120009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:59.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 -- 172.21.15.116:0/570244470 shutdown_connections 2023-11-08T08:21:59.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 --2- 172.21.15.116:0/570244470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300b0540 0x7fb1300c9bf0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 --2- 172.21.15.116:0/570244470 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300ca310 0x7fb1300b0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 -- 172.21.15.116:0/570244470 >> 172.21.15.116:0/570244470 conn(0x7fb1300cb540 msgr2=0x7fb1300cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 -- 172.21.15.116:0/570244470 shutdown_connections 2023-11-08T08:21:59.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.436+0000 7fb135403700 1 -- 172.21.15.116:0/570244470 wait complete. 2023-11-08T08:21:59.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.437+0000 7fb135403700 1 Processor -- start 2023-11-08T08:21:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 -- start start 2023-11-08T08:21:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300b0540 0x7fb13015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 0x7fb13015e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13015e910 0x7fb13014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13015bd90 con 0x7fb1300ca310 2023-11-08T08:21:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130159360 con 0x7fb13015e910 2023-11-08T08:21:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.438+0000 7fb135403700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13015b110 con 0x7fb1300b0540 2023-11-08T08:21:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 0x7fb13015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 0x7fb13015e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54086/0 (socket says 172.21.15.116:54086) 2023-11-08T08:21:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 -- 172.21.15.116:0/4188997637 learned_addr learned my addr 172.21.15.116:0/4188997637 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12effd700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300b0540 0x7fb13015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12f7fe700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13015e910 0x7fb13014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 -- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300b0540 msgr2=0x7fb13015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300b0540 0x7fb13015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 -- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13015e910 msgr2=0x7fb13014eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13015e910 0x7fb13014eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 -- 172.21.15.116:0/4188997637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200094a0 con 0x7fb1300ca310 2023-11-08T08:21:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12effd700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300b0540 0x7fb13015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12f7fe700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13015e910 0x7fb13014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:21:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb12e7fc700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 0x7fb13015e020 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb12400e810 tx=0x7fb12400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb124013610 con 0x7fb1300ca310 2023-11-08T08:21:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb124013770 con 0x7fb1300ca310 2023-11-08T08:21:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1300c9110 con 0x7fb1300ca310 2023-11-08T08:21:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb124013b20 con 0x7fb1300ca310 2023-11-08T08:21:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.439+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1300cacf0 con 0x7fb1300ca310 2023-11-08T08:21:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.440+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1300c93f0 con 0x7fb1300ca310 2023-11-08T08:21:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.441+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb124023070 con 0x7fb1300ca310 2023-11-08T08:21:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.441+0000 7fb1177fe700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb11c0682f0 0x7fb11c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.441+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fb124085710 con 0x7fb1300ca310 2023-11-08T08:21:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.441+0000 7fb12effd700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb11c0682f0 0x7fb11c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.441+0000 7fb12effd700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb11c0682f0 0x7fb11c06a7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb120000c00 tx=0x7fb12001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.445+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb1240521f0 con 0x7fb1300ca310 2023-11-08T08:21:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.631+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fb13015bed0 con 0x7fb11c0682f0 2023-11-08T08:21:59.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.631+0000 7fb1177fe700 1 -- 172.21.15.116:0/4188997637 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb13015bed0 con 0x7fb11c0682f0 2023-11-08T08:21:59.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb11c0682f0 msgr2=0x7fb11c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb11c0682f0 0x7fb11c06a7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb120000c00 tx=0x7fb12001c040 comp rx=0 tx=0).stop 2023-11-08T08:21:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 msgr2=0x7fb13015e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 0x7fb13015e020 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb12400e810 tx=0x7fb12400ebd0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 shutdown_connections 2023-11-08T08:21:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fb11c0682f0 0x7fb11c06a7a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13015e910 0x7fb13014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb1300b0540 0x7fb13015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 --2- 172.21.15.116:0/4188997637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1300ca310 0x7fb13015e020 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 >> 172.21.15.116:0/4188997637 conn(0x7fb1300cb540 msgr2=0x7fb1300af380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 shutdown_connections 2023-11-08T08:21:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.633+0000 7fb135403700 1 -- 172.21.15.116:0/4188997637 wait complete. 2023-11-08T08:21:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 5 to deep-scrub 2023-11-08T08:21:59.646 INFO:tasks.ceph:Scrubbing osd.6 2023-11-08T08:21:59.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:21:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.769+0000 7f2a277ef700 1 Processor -- start 2023-11-08T08:21:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.770+0000 7f2a277ef700 1 -- start start 2023-11-08T08:21:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.770+0000 7f2a277ef700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a2015c620 0x7f2a200cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.770+0000 7f2a277ef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 0x7f2a20160920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.770+0000 7f2a277ef700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a200cb160 con 0x7f2a20160e60 2023-11-08T08:21:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.770+0000 7f2a277ef700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a20066620 con 0x7f2a2015c620 2023-11-08T08:21:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.770+0000 7f2a277ef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a200cb420 con 0x7f2a200ae7a0 2023-11-08T08:21:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a2015c620 0x7f2a200cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a25d8c700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2a20160e60 0x7f2a200ccf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58660/0 (socket says 172.21.15.116:58660) 2023-11-08T08:21:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a2015c620 0x7f2a200cac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54088/0 (socket says 172.21.15.116:54088) 2023-11-08T08:21:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a25d8c700 1 -- 172.21.15.116:0/1661440267 learned_addr learned my addr 172.21.15.116:0/1661440267 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:21:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a24d8a700 1 --2- 172.21.15.116:0/1661440267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 0x7f2a20160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 -- 172.21.15.116:0/1661440267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 msgr2=0x7f2a20160920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 --2- 172.21.15.116:0/1661440267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 0x7f2a20160920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 -- 172.21.15.116:0/1661440267 >> v1:172.21.15.116:6789/0 conn(0x7f2a20160e60 legacy=0x7f2a200ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 -- 172.21.15.116:0/1661440267 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a200cb700 con 0x7f2a2015c620 2023-11-08T08:21:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a17fff700 1 -- 172.21.15.116:0/1661440267 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124585490 0 0) 0x7f2a200cb160 con 0x7f2a20160e60 2023-11-08T08:21:59.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a2558b700 1 --2- 172.21.15.116:0/1661440267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a200cac20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009840 tx=0x7f2a10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7bb2c7d441245a server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a17fff700 1 -- 172.21.15.116:0/1661440267 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10011830 con 0x7f2a2015c620 2023-11-08T08:21:59.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a17fff700 1 -- 172.21.15.116:0/1661440267 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a10011990 con 0x7f2a2015c620 2023-11-08T08:21:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.771+0000 7f2a17fff700 1 -- 172.21.15.116:0/1661440267 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10011d20 con 0x7f2a2015c620 2023-11-08T08:21:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 -- 172.21.15.116:0/1661440267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 msgr2=0x7f2a200cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 --2- 172.21.15.116:0/1661440267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a200cac20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009840 tx=0x7f2a10009b50 comp rx=0 tx=0).stop 2023-11-08T08:21:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 -- 172.21.15.116:0/1661440267 shutdown_connections 2023-11-08T08:21:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 --2- 172.21.15.116:0/1661440267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a200cac20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 --2- 172.21.15.116:0/1661440267 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 0x7f2a20160920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 -- 172.21.15.116:0/1661440267 >> 172.21.15.116:0/1661440267 conn(0x7f2a200cf370 msgr2=0x7f2a200cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 -- 172.21.15.116:0/1661440267 shutdown_connections 2023-11-08T08:21:59.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.772+0000 7f2a277ef700 1 -- 172.21.15.116:0/1661440267 wait complete. 2023-11-08T08:21:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.773+0000 7f2a277ef700 1 Processor -- start 2023-11-08T08:21:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.773+0000 7f2a277ef700 1 -- start start 2023-11-08T08:21:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a277ef700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 0x7f2a201f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a277ef700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a201585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a277ef700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a20160e60 0x7f2a201f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a277ef700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20158e00 con 0x7f2a2015c620 2023-11-08T08:21:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a277ef700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20158b30 con 0x7f2a20160e60 2023-11-08T08:21:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a277ef700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a200b2310 con 0x7f2a200ae7a0 2023-11-08T08:21:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a24d8a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a201585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a24d8a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a201585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54096/0 (socket says 172.21.15.116:54096) 2023-11-08T08:21:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.774+0000 7f2a24d8a700 1 -- 172.21.15.116:0/2100933505 learned_addr learned my addr 172.21.15.116:0/2100933505 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a24d8a700 1 -- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 msgr2=0x7f2a201f2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a24d8a700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2a200ae7a0 0x7f2a201f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a24d8a700 1 -- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a20160e60 msgr2=0x7f2a201f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a25d8c700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a20160e60 0x7f2a201f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a24d8a700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a20160e60 0x7f2a201f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a24d8a700 1 -- 172.21.15.116:0/2100933505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a100094a0 con 0x7f2a2015c620 2023-11-08T08:21:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a24d8a700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a201585f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e810 tx=0x7f2a1c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c013610 con 0x7f2a2015c620 2023-11-08T08:21:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c013770 con 0x7f2a2015c620 2023-11-08T08:21:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a20160400 con 0x7f2a2015c620 2023-11-08T08:21:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c013b20 con 0x7f2a2015c620 2023-11-08T08:21:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.775+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a200cb9e0 con 0x7f2a2015c620 2023-11-08T08:21:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.776+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a200cd650 con 0x7f2a2015c620 2023-11-08T08:21:59.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a1c023070 con 0x7f2a2015c620 2023-11-08T08:21:59.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a15ffb700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a040682f0 0x7f2a0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f2a1c085670 con 0x7f2a2015c620 2023-11-08T08:21:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a2558b700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a040682f0 0x7f2a0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a15ffb700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7f2a0406e760 0x7f2a04070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 --> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a04071240 con 0x7f2a0406e760 2023-11-08T08:21:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a1c087d20 con 0x7f2a2015c620 2023-11-08T08:21:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a25d8c700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7f2a0406e760 0x7f2a04070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.777+0000 7f2a2558b700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a040682f0 0x7f2a0406a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2a100090b0 tx=0x7f2a10010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.778+0000 7f2a25d8c700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7f2a0406e760 0x7f2a04070b70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.779+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== osd.6 v2:172.21.15.129:6824/121459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2a04071240 con 0x7f2a0406e760 2023-11-08T08:21:59.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.792+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 --> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2a200743f0 con 0x7f2a0406e760 2023-11-08T08:21:59.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.792+0000 7f2a15ffb700 1 -- 172.21.15.116:0/2100933505 <== osd.6 v2:172.21.15.129:6824/121459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2a200743f0 con 0x7f2a0406e760 2023-11-08T08:21:59.795 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:21:59.795 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:21:59.796 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:21:59.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7f2a0406e760 msgr2=0x7f2a04070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.129:6824/121459,v1:172.21.15.129:6825/121459] conn(0x7f2a0406e760 0x7f2a04070b70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a040682f0 msgr2=0x7f2a0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f2a040682f0 0x7f2a0406a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2a100090b0 tx=0x7f2a10010000 comp rx=0 tx=0).stop 2023-11-08T08:21:59.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 msgr2=0x7f2a201585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 --2- 172.21.15.116:0/2100933505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a2015c620 0x7f2a201585f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e810 tx=0x7f2a1c00ebd0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a25d8c700 1 -- 172.21.15.116:0/2100933505 reap_dead start 2023-11-08T08:21:59.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 shutdown_connections 2023-11-08T08:21:59.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 >> 172.21.15.116:0/2100933505 conn(0x7f2a200cf370 msgr2=0x7f2a2015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 shutdown_connections 2023-11-08T08:21:59.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.793+0000 7f2a277ef700 1 -- 172.21.15.116:0/2100933505 wait complete. 2023-11-08T08:21:59.805 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-11-08T08:21:59.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.925+0000 7f8272cf1700 1 Processor -- start 2023-11-08T08:21:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.925+0000 7f8272cf1700 1 -- start start 2023-11-08T08:21:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8272cf1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8272cf1700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f826c06a980 0x7f826c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8272cf1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c15cc10 con 0x7f826c071890 2023-11-08T08:21:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8272cf1700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c06c9b0 con 0x7f826c06a980 2023-11-08T08:21:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8272cf1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c06cc90 con 0x7f826c0cf710 2023-11-08T08:21:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f826c06a980 0x7f826c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8270a8d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f826c06a980 0x7f826c071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:53004/0 (socket says 172.21.15.116:53004) 2023-11-08T08:21:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8270a8d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c0ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:50282/0 (socket says 172.21.15.116:50282) 2023-11-08T08:21:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 -- 172.21.15.116:0/460277109 learned_addr learned my addr 172.21.15.116:0/460277109 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f826bfff700 1 --1- 172.21.15.116:0/460277109 >> v1:172.21.15.129:6789/0 conn(0x7f826c071890 0x7f826c06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:47400/0 (socket says 172.21.15.116:47400) 2023-11-08T08:21:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 -- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 msgr2=0x7f826c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:21:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 --2- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c0ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 -- 172.21.15.116:0/460277109 >> v1:172.21.15.129:6789/0 conn(0x7f826c071890 legacy=0x7f826c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f827128e700 1 -- 172.21.15.116:0/460277109 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c06cf70 con 0x7f826c06a980 2023-11-08T08:21:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.926+0000 7f8270a8d700 1 --2- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:21:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f827128e700 1 --2- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c06a980 0x7f826c071220 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8264009e50 tx=0x7f826400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=660b4484c3534cb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f826b7fe700 1 -- 172.21.15.116:0/460277109 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400e070 con 0x7f826c06a980 2023-11-08T08:21:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f826b7fe700 1 -- 172.21.15.116:0/460277109 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826400b910 con 0x7f826c06a980 2023-11-08T08:21:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f826b7fe700 1 -- 172.21.15.116:0/460277109 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400bcc0 con 0x7f826c06a980 2023-11-08T08:21:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 -- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c06a980 msgr2=0x7f826c071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 --2- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c06a980 0x7f826c071220 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8264009e50 tx=0x7f826400b330 comp rx=0 tx=0).stop 2023-11-08T08:21:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 -- 172.21.15.116:0/460277109 shutdown_connections 2023-11-08T08:21:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 --2- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c06a980 0x7f826c071220 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 --2- 172.21.15.116:0/460277109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 -- 172.21.15.116:0/460277109 >> 172.21.15.116:0/460277109 conn(0x7f826c06c010 msgr2=0x7f826c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:21:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 -- 172.21.15.116:0/460277109 shutdown_connections 2023-11-08T08:21:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.927+0000 7f8272cf1700 1 -- 172.21.15.116:0/460277109 wait complete. 2023-11-08T08:21:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.929+0000 7f8272cf1700 1 Processor -- start 2023-11-08T08:21:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.929+0000 7f8272cf1700 1 -- start start 2023-11-08T08:21:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8272cf1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 0x7f826c1f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8272cf1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c071890 0x7f826c1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8272cf1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c15e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8272cf1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c1587e0 con 0x7f826c06a980 2023-11-08T08:21:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8272cf1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c1f3880 con 0x7f826c071890 2023-11-08T08:21:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8272cf1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c1f2fe0 con 0x7f826c0cf710 2023-11-08T08:21:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 0x7f826c1f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 0x7f826c1f28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54106/0 (socket says 172.21.15.116:54106) 2023-11-08T08:21:59.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 -- 172.21.15.116:0/4247996386 learned_addr learned my addr 172.21.15.116:0/4247996386 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:21:59.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f827128e700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c15e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f826bfff700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c071890 0x7f826c1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 -- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 msgr2=0x7f826c15e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c15e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 -- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c071890 msgr2=0x7f826c1582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:21:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c071890 0x7f826c1582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:21:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.930+0000 7f8270a8d700 1 -- 172.21.15.116:0/4247996386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82600094a0 con 0x7f826c06a980 2023-11-08T08:21:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f826bfff700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c071890 0x7f826c1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f827128e700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c15e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:21:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f8270a8d700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 0x7f826c1f28e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f826000ebe0 tx=0x7f8260009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826000c8b0 con 0x7f826c06a980 2023-11-08T08:21:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826000ca10 con 0x7f826c06a980 2023-11-08T08:21:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826000cda0 con 0x7f826c06a980 2023-11-08T08:21:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8264009b30 con 0x7f826c06a980 2023-11-08T08:21:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.931+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c06d530 con 0x7f826c06a980 2023-11-08T08:21:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.932+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c06d250 con 0x7f826c06a980 2023-11-08T08:21:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.932+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8260019030 con 0x7f826c06a980 2023-11-08T08:21:59.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.932+0000 7f82697fa700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82500682f0 0x7f825006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:21:59.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.933+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f8260085a00 con 0x7f826c06a980 2023-11-08T08:21:59.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.933+0000 7f826bfff700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82500682f0 0x7f825006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:21:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.933+0000 7f826bfff700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82500682f0 0x7f825006a7a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f826c157710 tx=0x7f8258009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:21:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:21:59.937+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8260052560 con 0x7f826c06a980 2023-11-08T08:22:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.116+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f826c15d2a0 con 0x7f82500682f0 2023-11-08T08:22:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.116+0000 7f82697fa700 1 -- 172.21.15.116:0/4247996386 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f826c15d2a0 con 0x7f82500682f0 2023-11-08T08:22:00.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82500682f0 msgr2=0x7f825006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82500682f0 0x7f825006a7a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f826c157710 tx=0x7f8258009040 comp rx=0 tx=0).stop 2023-11-08T08:22:00.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 msgr2=0x7f826c1f28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 0x7f826c1f28e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f826000ebe0 tx=0x7f8260009e20 comp rx=0 tx=0).stop 2023-11-08T08:22:00.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 shutdown_connections 2023-11-08T08:22:00.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f826c0cf710 0x7f826c15e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826c06a980 0x7f826c1f28e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f82500682f0 0x7f825006a7a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 --2- 172.21.15.116:0/4247996386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c071890 0x7f826c1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 >> 172.21.15.116:0/4247996386 conn(0x7f826c06c010 msgr2=0x7f826c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 shutdown_connections 2023-11-08T08:22:00.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.118+0000 7f8272cf1700 1 -- 172.21.15.116:0/4247996386 wait complete. 2023-11-08T08:22:00.123 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 6 to deep-scrub 2023-11-08T08:22:00.131 INFO:tasks.ceph:Scrubbing osd.7 2023-11-08T08:22:00.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-11-08T08:22:00.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.254+0000 7f6895c52700 1 Processor -- start 2023-11-08T08:22:00.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.254+0000 7f6895c52700 1 -- start start 2023-11-08T08:22:00.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f6895c52700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f689015e070 0x7f689006b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f6895c52700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f689015a860 0x7f689006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f6895c52700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900cea10 con 0x7f68900ceea0 2023-11-08T08:22:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f6895c52700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689015ae80 con 0x7f689015e070 2023-11-08T08:22:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f6895c52700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68900cbd10 con 0x7f689015a860 2023-11-08T08:22:00.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688f7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f689015e070 0x7f689006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688effd700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f68900ceea0 0x7f689015a150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58662/0 (socket says 172.21.15.116:58662) 2023-11-08T08:22:00.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688effd700 1 -- 172.21.15.116:0/2677302117 learned_addr learned my addr 172.21.15.116:0/2677302117 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:22:00.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688ffff700 1 --2- 172.21.15.116:0/2677302117 >> v2:172.21.15.129:3300/0 conn(0x7f689015a860 0x7f689006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688f7fe700 1 -- 172.21.15.116:0/2677302117 >> v2:172.21.15.129:3300/0 conn(0x7f689015a860 msgr2=0x7f689006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:00.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688f7fe700 1 --2- 172.21.15.116:0/2677302117 >> v2:172.21.15.129:3300/0 conn(0x7f689015a860 0x7f689006b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688f7fe700 1 -- 172.21.15.116:0/2677302117 >> v1:172.21.15.116:6789/0 conn(0x7f68900ceea0 legacy=0x7f689015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688e7fc700 1 -- 172.21.15.116:0/2677302117 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935435476 0 0) 0x7f68900cea10 con 0x7f68900ceea0 2023-11-08T08:22:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688f7fe700 1 -- 172.21.15.116:0/2677302117 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68900ce0e0 con 0x7f689015e070 2023-11-08T08:22:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688f7fe700 1 --2- 172.21.15.116:0/2677302117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f689006b380 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f68900709c0 tx=0x7f6884009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08b2b943f8a58b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688e7fc700 1 -- 172.21.15.116:0/2677302117 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884011830 con 0x7f689015e070 2023-11-08T08:22:00.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688e7fc700 1 -- 172.21.15.116:0/2677302117 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6884011990 con 0x7f689015e070 2023-11-08T08:22:00.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.255+0000 7f688e7fc700 1 -- 172.21.15.116:0/2677302117 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884011d20 con 0x7f689015e070 2023-11-08T08:22:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 -- 172.21.15.116:0/2677302117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 msgr2=0x7f689006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 --2- 172.21.15.116:0/2677302117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f689006b380 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f68900709c0 tx=0x7f6884009b20 comp rx=0 tx=0).stop 2023-11-08T08:22:00.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 -- 172.21.15.116:0/2677302117 shutdown_connections 2023-11-08T08:22:00.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 --2- 172.21.15.116:0/2677302117 >> v2:172.21.15.129:3300/0 conn(0x7f689015a860 0x7f689006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 --2- 172.21.15.116:0/2677302117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f689006b380 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 -- 172.21.15.116:0/2677302117 >> 172.21.15.116:0/2677302117 conn(0x7f68900cfc30 msgr2=0x7f68900d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 -- 172.21.15.116:0/2677302117 shutdown_connections 2023-11-08T08:22:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.256+0000 7f6895c52700 1 -- 172.21.15.116:0/2677302117 wait complete. 2023-11-08T08:22:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.257+0000 7f6895c52700 1 Processor -- start 2023-11-08T08:22:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.257+0000 7f6895c52700 1 -- start start 2023-11-08T08:22:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f6895c52700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68900ceea0 0x7f68900b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f6895c52700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f689015a860 0x7f68901f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f6895c52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f68900c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f6895c52700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900b3d10 con 0x7f689015e070 2023-11-08T08:22:00.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f6895c52700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900b3f40 con 0x7f68900ceea0 2023-11-08T08:22:00.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f6895c52700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68900b4220 con 0x7f689015a860 2023-11-08T08:22:00.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f68900c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f68900c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54126/0 (socket says 172.21.15.116:54126) 2023-11-08T08:22:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 -- 172.21.15.116:0/2996252585 learned_addr learned my addr 172.21.15.116:0/2996252585 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 -- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f689015a860 msgr2=0x7f68901f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f689015a860 0x7f68901f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 -- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68900ceea0 msgr2=0x7f68900b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68900ceea0 0x7f68900b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.258+0000 7f688ffff700 1 -- 172.21.15.116:0/2996252585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68840094a0 con 0x7f689015e070 2023-11-08T08:22:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.259+0000 7f688ffff700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f68900c5e90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f688800cd70 tx=0x7f688800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.259+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888003a20 con 0x7f689015e070 2023-11-08T08:22:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.259+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6888003b80 con 0x7f689015e070 2023-11-08T08:22:00.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.259+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888004020 con 0x7f689015e070 2023-11-08T08:22:00.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.259+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68900ce6a0 con 0x7f689015e070 2023-11-08T08:22:00.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.259+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68900ce3c0 con 0x7f689015e070 2023-11-08T08:22:00.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.260+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6890150080 con 0x7f689015e070 2023-11-08T08:22:00.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.260+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f688801c020 con 0x7f689015e070 2023-11-08T08:22:00.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.260+0000 7f6877fff700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f687c0682f0 0x7f687c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f6888012070 con 0x7f689015e070 2023-11-08T08:22:00.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f688f7fe700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f687c0682f0 0x7f687c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f6877fff700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7f687c06e760 0x7f687c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f687c071240 con 0x7f687c06e760 2023-11-08T08:22:00.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68880886b0 con 0x7f689015e070 2023-11-08T08:22:00.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f688effd700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7f687c06e760 0x7f687c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f688f7fe700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f687c0682f0 0x7f687c06a7a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6884000c00 tx=0x7f6884010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.261+0000 7f688effd700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7f687c06e760 0x7f687c070b70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.262+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== osd.7 v2:172.21.15.129:6816/121422 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f687c071240 con 0x7f687c06e760 2023-11-08T08:22:00.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.276+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 --> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f68900743f0 con 0x7f687c06e760 2023-11-08T08:22:00.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6877fff700 1 -- 172.21.15.116:0/2996252585 <== osd.7 v2:172.21.15.129:6816/121422 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f68900743f0 con 0x7f687c06e760 2023-11-08T08:22:00.279 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-08T08:22:00.280 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T08:22:00.280 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-08T08:22:00.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7f687c06e760 msgr2=0x7f687c070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.129:6816/121422,v1:172.21.15.129:6817/121422] conn(0x7f687c06e760 0x7f687c070b70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f687c0682f0 msgr2=0x7f687c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f687c0682f0 0x7f687c06a7a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6884000c00 tx=0x7f6884010000 comp rx=0 tx=0).stop 2023-11-08T08:22:00.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 msgr2=0x7f68900c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 --2- 172.21.15.116:0/2996252585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f689015e070 0x7f68900c5e90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f688800cd70 tx=0x7f688800a570 comp rx=0 tx=0).stop 2023-11-08T08:22:00.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f688ffff700 1 -- 172.21.15.116:0/2996252585 reap_dead start 2023-11-08T08:22:00.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 shutdown_connections 2023-11-08T08:22:00.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 >> 172.21.15.116:0/2996252585 conn(0x7f68900cfc30 msgr2=0x7f6890070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 shutdown_connections 2023-11-08T08:22:00.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.277+0000 7f6895c52700 1 -- 172.21.15.116:0/2996252585 wait complete. 2023-11-08T08:22:00.290 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-11-08T08:22:00.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.410+0000 7f4776843700 1 Processor -- start 2023-11-08T08:22:00.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.410+0000 7f4776843700 1 -- start start 2023-11-08T08:22:00.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4776843700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f477006a980 0x7f47700ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4776843700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4770071890 0x7f4770071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4776843700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477015cc10 con 0x7f47700cf710 2023-11-08T08:22:00.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4776843700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477006c9b0 con 0x7f4770071890 2023-11-08T08:22:00.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4776843700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477006cc90 con 0x7f477006a980 2023-11-08T08:22:00.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f476f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f477006a980 0x7f47700ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4774de0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4770071890 0x7f4770071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4774de0700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4770071890 0x7f4770071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.116:53046/0 (socket says 172.21.15.116:53046) 2023-11-08T08:22:00.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f476f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f477006a980 0x7f47700ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:50300/0 (socket says 172.21.15.116:50300) 2023-11-08T08:22:00.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f4774de0700 1 -- 172.21.15.116:0/3163718875 learned_addr learned my addr 172.21.15.116:0/3163718875 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:00.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.411+0000 7f476ffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f47700cf710 0x7f477006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:47412/0 (socket says 172.21.15.116:47412) 2023-11-08T08:22:00.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f4774de0700 1 -- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f477006a980 msgr2=0x7f47700ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:00.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f4774de0700 1 --2- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f477006a980 0x7f47700ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f4774de0700 1 -- 172.21.15.116:0/3163718875 >> v1:172.21.15.129:6789/0 conn(0x7f47700cf710 legacy=0x7f477006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f4774de0700 1 -- 172.21.15.116:0/3163718875 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477006cf70 con 0x7f4770071890 2023-11-08T08:22:00.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f476effd700 1 -- 172.21.15.116:0/3163718875 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407730881 0 0) 0x7f477015cc10 con 0x7f47700cf710 2023-11-08T08:22:00.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f4774de0700 1 --2- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4770071890 0x7f4770071220 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4768009e50 tx=0x7f476800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=668908d4c322fc58 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f476effd700 1 -- 172.21.15.116:0/3163718875 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476800e070 con 0x7f4770071890 2023-11-08T08:22:00.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f476effd700 1 -- 172.21.15.116:0/3163718875 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476800b910 con 0x7f4770071890 2023-11-08T08:22:00.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.412+0000 7f476effd700 1 -- 172.21.15.116:0/3163718875 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476800bcc0 con 0x7f4770071890 2023-11-08T08:22:00.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 -- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4770071890 msgr2=0x7f4770071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 --2- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4770071890 0x7f4770071220 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4768009e50 tx=0x7f476800b330 comp rx=0 tx=0).stop 2023-11-08T08:22:00.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 -- 172.21.15.116:0/3163718875 shutdown_connections 2023-11-08T08:22:00.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 --2- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4770071890 0x7f4770071220 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 --2- 172.21.15.116:0/3163718875 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f477006a980 0x7f47700ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 -- 172.21.15.116:0/3163718875 >> 172.21.15.116:0/3163718875 conn(0x7f477006c010 msgr2=0x7f47700cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 -- 172.21.15.116:0/3163718875 shutdown_connections 2023-11-08T08:22:00.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.413+0000 7f4776843700 1 -- 172.21.15.116:0/3163718875 wait complete. 2023-11-08T08:22:00.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.414+0000 7f4776843700 1 Processor -- start 2023-11-08T08:22:00.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.414+0000 7f4776843700 1 -- start start 2023-11-08T08:22:00.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4776843700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477006a980 0x7f477015a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4776843700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4770071890 0x7f47700cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4776843700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 0x7f47700b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4776843700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47700c6ca0 con 0x7f47700cf710 2023-11-08T08:22:00.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4776843700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47700b43b0 con 0x7f477006a980 2023-11-08T08:22:00.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4776843700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47700b35c0 con 0x7f4770071890 2023-11-08T08:22:00.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 0x7f47700b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 0x7f47700b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54132/0 (socket says 172.21.15.116:54132) 2023-11-08T08:22:00.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 -- 172.21.15.116:0/1314404068 learned_addr learned my addr 172.21.15.116:0/1314404068 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:00.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f476ffff700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477006a980 0x7f477015a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f476f7fe700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4770071890 0x7f47700cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 -- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4770071890 msgr2=0x7f47700cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4770071890 0x7f47700cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 -- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477006a980 msgr2=0x7f477015a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477006a980 0x7f477015a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 -- 172.21.15.116:0/1314404068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4768009b30 con 0x7f47700cf710 2023-11-08T08:22:00.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f476f7fe700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4770071890 0x7f47700cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-08T08:22:00.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f4774de0700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 0x7f47700b2a00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f476800b680 tx=0x7f4768004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768004960 con 0x7f47700cf710 2023-11-08T08:22:00.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.415+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47680099b0 con 0x7f47700cf710 2023-11-08T08:22:00.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.416+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477006d530 con 0x7f47700cf710 2023-11-08T08:22:00.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.416+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476801c660 con 0x7f47700cf710 2023-11-08T08:22:00.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.416+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477006d250 con 0x7f47700cf710 2023-11-08T08:22:00.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.416+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4770160480 con 0x7f47700cf710 2023-11-08T08:22:00.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.417+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f476802c070 con 0x7f47700cf710 2023-11-08T08:22:00.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.417+0000 7f476cff9700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f475c0682f0 0x7f475c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.417+0000 7f476ffff700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f475c0682f0 0x7f475c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.417+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f47680874d0 con 0x7f47700cf710 2023-11-08T08:22:00.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.418+0000 7f476ffff700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f475c0682f0 0x7f475c06a7a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4764000f60 tx=0x7f4764007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.422+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4768053fd0 con 0x7f47700cf710 2023-11-08T08:22:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.604+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f47700743f0 con 0x7f475c0682f0 2023-11-08T08:22:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.604+0000 7f476cff9700 1 -- 172.21.15.116:0/1314404068 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f47700743f0 con 0x7f475c0682f0 2023-11-08T08:22:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f475c0682f0 msgr2=0x7f475c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f475c0682f0 0x7f475c06a7a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4764000f60 tx=0x7f4764007e70 comp rx=0 tx=0).stop 2023-11-08T08:22:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 msgr2=0x7f47700b2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 0x7f47700b2a00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f476800b680 tx=0x7f4768004ad0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 shutdown_connections 2023-11-08T08:22:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47700cf710 0x7f47700b2a00 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f475c0682f0 0x7f475c06a7a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477006a980 0x7f477015a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 --2- 172.21.15.116:0/1314404068 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4770071890 0x7f47700cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 >> 172.21.15.116:0/1314404068 conn(0x7f477006c010 msgr2=0x7f477015ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 shutdown_connections 2023-11-08T08:22:00.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.606+0000 7f4776843700 1 -- 172.21.15.116:0/1314404068 wait complete. 2023-11-08T08:22:00.611 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 7 to deep-scrub 2023-11-08T08:22:00.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:22:00.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.738+0000 7f8029bca700 1 Processor -- start 2023-11-08T08:22:00.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.738+0000 7f8029bca700 1 -- start start 2023-11-08T08:22:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f8029bca700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f802406a270 0x7f802406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f8029bca700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f8024070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f8029bca700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802406ab80 con 0x7f802415cc80 2023-11-08T08:22:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f8029bca700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80240b2490 con 0x7f802406a270 2023-11-08T08:22:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f8029bca700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80240b1810 con 0x7f802406e570 2023-11-08T08:22:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f80237fe700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f802406a270 0x7f802406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.739+0000 7f8023fff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f8024070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f8023fff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f8024070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.116:50310/0 (socket says 172.21.15.116:50310) 2023-11-08T08:22:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f8022ffd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f802415cc80 0x7f802406de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.116:47422/0 (socket says 172.21.15.116:47422) 2023-11-08T08:22:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f8023fff700 1 -- 172.21.15.116:0/3034290812 learned_addr learned my addr 172.21.15.116:0/3034290812 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80237fe700 1 -- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 msgr2=0x7f8024070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80237fe700 1 --2- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f8024070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80237fe700 1 -- 172.21.15.116:0/3034290812 >> v1:172.21.15.129:6789/0 conn(0x7f802415cc80 legacy=0x7f802406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80227fc700 1 -- 172.21.15.116:0/3034290812 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817634795 0 0) 0x7f802406ab80 con 0x7f802415cc80 2023-11-08T08:22:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80237fe700 1 -- 172.21.15.116:0/3034290812 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80240b1af0 con 0x7f802406a270 2023-11-08T08:22:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f8023fff700 1 --2- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f8024070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-08T08:22:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80237fe700 1 --2- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802406a270 0x7f802406a640 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8018009840 tx=0x7f8018009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de734324c8b0cc5d server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80227fc700 1 -- 172.21.15.116:0/3034290812 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018011830 con 0x7f802406a270 2023-11-08T08:22:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80227fc700 1 -- 172.21.15.116:0/3034290812 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8018011990 con 0x7f802406a270 2023-11-08T08:22:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.740+0000 7f80227fc700 1 -- 172.21.15.116:0/3034290812 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018011d20 con 0x7f802406a270 2023-11-08T08:22:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 -- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802406a270 msgr2=0x7f802406a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 --2- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802406a270 0x7f802406a640 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8018009840 tx=0x7f8018009b50 comp rx=0 tx=0).stop 2023-11-08T08:22:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 -- 172.21.15.116:0/3034290812 shutdown_connections 2023-11-08T08:22:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 --2- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802406a270 0x7f802406a640 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 --2- 172.21.15.116:0/3034290812 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f8024070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 -- 172.21.15.116:0/3034290812 >> 172.21.15.116:0/3034290812 conn(0x7f802406c010 msgr2=0x7f80240cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 -- 172.21.15.116:0/3034290812 shutdown_connections 2023-11-08T08:22:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.741+0000 7f8029bca700 1 -- 172.21.15.116:0/3034290812 wait complete. 2023-11-08T08:22:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.742+0000 7f8029bca700 1 Processor -- start 2023-11-08T08:22:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.742+0000 7f8029bca700 1 -- start start 2023-11-08T08:22:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8029bca700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 0x7f802414a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8029bca700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f802415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8029bca700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802415cc80 0x7f802415f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8029bca700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80240c87c0 con 0x7f802406a270 2023-11-08T08:22:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8029bca700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80240b5aa0 con 0x7f802415cc80 2023-11-08T08:22:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8029bca700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802414a900 con 0x7f802406e570 2023-11-08T08:22:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 0x7f802414a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 0x7f802414a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54148/0 (socket says 172.21.15.116:54148) 2023-11-08T08:22:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 -- 172.21.15.116:0/2889335617 learned_addr learned my addr 172.21.15.116:0/2889335617 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8023fff700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802415cc80 0x7f802415f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8022ffd700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f802415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 -- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 msgr2=0x7f802415ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f802415ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 -- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802415cc80 msgr2=0x7f802415f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802415cc80 0x7f802415f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 -- 172.21.15.116:0/2889335617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c009b30 con 0x7f802406a270 2023-11-08T08:22:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f8022ffd700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f802415ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-08T08:22:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.743+0000 7f80237fe700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 0x7f802414a3c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8018009fd0 tx=0x7f8018004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.744+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80180036a0 con 0x7f802406a270 2023-11-08T08:22:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.744+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801801e070 con 0x7f802406a270 2023-11-08T08:22:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.744+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80180094a0 con 0x7f802406a270 2023-11-08T08:22:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.744+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018003a60 con 0x7f802406a270 2023-11-08T08:22:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.744+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80240b20b0 con 0x7f802406a270 2023-11-08T08:22:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.745+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80240b1dd0 con 0x7f802406a270 2023-11-08T08:22:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.745+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f801800f050 con 0x7f802406a270 2023-11-08T08:22:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.745+0000 7f800bfff700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80100682f0 0x7f801006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.745+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f8018086990 con 0x7f802406a270 2023-11-08T08:22:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.745+0000 7f8022ffd700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80100682f0 0x7f801006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.746+0000 7f8022ffd700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80100682f0 0x7f801006a7a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8014007fd0 tx=0x7f8014009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.750+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80180534f0 con 0x7f802406a270 2023-11-08T08:22:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.924+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f802415daa0 con 0x7f80100682f0 2023-11-08T08:22:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.926+0000 7f800bfff700 1 -- 172.21.15.116:0/2889335617 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72974 (secure 0 0 0) 0x7f802415daa0 con 0x7f80100682f0 2023-11-08T08:22:00.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:22:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80100682f0 msgr2=0x7f801006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80100682f0 0x7f801006a7a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8014007fd0 tx=0x7f8014009040 comp rx=0 tx=0).stop 2023-11-08T08:22:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 msgr2=0x7f802414a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 0x7f802414a3c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8018009fd0 tx=0x7f8018004750 comp rx=0 tx=0).stop 2023-11-08T08:22:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 shutdown_connections 2023-11-08T08:22:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7f80100682f0 0x7f801006a7a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f802415cc80 0x7f802415f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802406a270 0x7f802414a3c0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 --2- 172.21.15.116:0/2889335617 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f802406e570 0x7f802415ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:00.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.928+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 >> 172.21.15.116:0/2889335617 conn(0x7f802406c010 msgr2=0x7f80240afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:00.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.929+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 shutdown_connections 2023-11-08T08:22:00.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:00.929+0000 7f8029bca700 1 -- 172.21.15.116:0/2889335617 wait complete. 2023-11-08T08:22:00.934 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:22:00.942 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2023-11-08T08:22:00.396387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":79204,"kb_used_data":6756,"kb_used_omap":0,"kb_used_meta":72384,"kb_avail":754895516,"statfs":{"total":773094113280,"available":773013008384,"internally_reserved":0,"allocated":6918144,"data_stored":3670651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74121216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.805001"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:52.679003+0000","last_change":"2023-11-08T08:18:37.486036+0000","last_active":"2023-11-08T08:21:52.679003+0000","last_peered":"2023-11-08T08:21:52.679003+0000","last_clean":"2023-11-08T08:21:52.679003+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:52.679003+0000","last_undegraded":"2023-11-08T08:21:52.679003+0000","last_fullsized":"2023-11-08T08:21:52.679003+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:30:40.187500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.595641+0000","last_change":"2023-11-08T08:18:37.461655+0000","last_active":"2023-11-08T08:21:51.595641+0000","last_peered":"2023-11-08T08:21:51.595641+0000","last_clean":"2023-11-08T08:21:51.595641+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:21:51.595641+0000","last_undegraded":"2023-11-08T08:21:51.595641+0000","last_fullsized":"2023-11-08T08:21:51.595641+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":45,"reported_epoch":57,"state":"active+clean","last_fresh":"2023-11-08T08:21:49.595251+0000","last_change":"2023-11-08T08:18:37.461310+0000","last_active":"2023-11-08T08:21:49.595251+0000","last_peered":"2023-11-08T08:21:49.595251+0000","last_clean":"2023-11-08T08:21:49.595251+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:21:49.595251+0000","last_undegraded":"2023-11-08T08:21:49.595251+0000","last_fullsized":"2023-11-08T08:21:49.595251+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":201,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.593171+0000","last_change":"2023-11-08T08:18:37.458217+0000","last_active":"2023-11-08T08:21:51.593171+0000","last_peered":"2023-11-08T08:21:51.593171+0000","last_clean":"2023-11-08T08:21:51.593171+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:21:51.593171+0000","last_undegraded":"2023-11-08T08:21:51.593171+0000","last_fullsized":"2023-11-08T08:21:51.593171+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T12:35:25.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":46,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.726953+0000","last_change":"2023-11-08T08:18:37.461366+0000","last_active":"2023-11-08T08:21:51.726953+0000","last_peered":"2023-11-08T08:21:51.726953+0000","last_clean":"2023-11-08T08:21:51.726953+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:21:51.726953+0000","last_undegraded":"2023-11-08T08:21:51.726953+0000","last_fullsized":"2023-11-08T08:21:51.726953+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T16:52:27.798714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.798165+0000","last_change":"2023-11-08T08:18:37.461028+0000","last_active":"2023-11-08T08:21:51.798165+0000","last_peered":"2023-11-08T08:21:51.798165+0000","last_clean":"2023-11-08T08:21:51.798165+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:51.798165+0000","last_undegraded":"2023-11-08T08:21:51.798165+0000","last_fullsized":"2023-11-08T08:21:51.798165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.593708+0000","last_change":"2023-11-08T08:18:37.485995+0000","last_active":"2023-11-08T08:21:51.593708+0000","last_peered":"2023-11-08T08:21:51.593708+0000","last_clean":"2023-11-08T08:21:51.593708+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:21:51.593708+0000","last_undegraded":"2023-11-08T08:21:51.593708+0000","last_fullsized":"2023-11-08T08:21:51.593708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.593865+0000","last_change":"2023-11-08T08:18:37.486004+0000","last_active":"2023-11-08T08:21:51.593865+0000","last_peered":"2023-11-08T08:21:51.593865+0000","last_clean":"2023-11-08T08:21:51.593865+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:21:51.593865+0000","last_undegraded":"2023-11-08T08:21:51.593865+0000","last_fullsized":"2023-11-08T08:21:51.593865+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:38:15.142978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":46,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-11-08T08:21:51.798434+0000","last_change":"2023-11-08T08:18:37.486055+0000","last_active":"2023-11-08T08:21:51.798434+0000","last_peered":"2023-11-08T08:21:51.798434+0000","last_clean":"2023-11-08T08:21:51.798434+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:21:51.798434+0000","last_undegraded":"2023-11-08T08:21:51.798434+0000","last_fullsized":"2023-11-08T08:21:51.798434+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:18:36.449394+0000","last_clean_scrub_stamp":"2023-11-08T08:18:36.449394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T10:50:13.194545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9412,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362428,"statfs":{"total":96636764160,"available":96627126272,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.37},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":1.611,"15min":1.611},"last":0.55800000000000005}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.67100000000000004,"15min":0.72499999999999998},"min":{"1min":0.35699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.73899999999999999,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.68200000000000005,"15min":0.73099999999999998},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81299999999999994,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.51800000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73499999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.47299999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55900000000000005,"15min":0.60299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59699999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57099999999999995,"15min":0.60099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58999999999999997}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61499999999999999,"5min":0.872,"15min":0.872},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.621}]},{"osd":6,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.5},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70699999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9600,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362240,"statfs":{"total":96636764160,"available":96626933760,"internally_reserved":0,"allocated":712704,"data_stored":310136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91200000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.115,"15min":1.115},"last":0.42299999999999999}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.44400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":1.089,"15min":1.089},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.996,"15min":0.996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.135},"last":0.496},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.66000000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.67300000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58199999999999996}]},{"osd":7,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.747,"15min":0.747},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9540,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362300,"statfs":{"total":96636764160,"available":96626995200,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.311,"15min":0.308},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.379,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.309},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.372}]},{"osd":2,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":1.149,"15min":1.149},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.443,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48399999999999999,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.40799999999999997}]},{"osd":4,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61099999999999999,"15min":0.56499999999999995},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70799999999999996,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.66200000000000003,"15min":0.63200000000000001},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.67700000000000005,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.65400000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77600000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67300000000000004,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.72799999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67600000000000005,"15min":0.629},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83199999999999996,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69299999999999995,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.68700000000000006}]}]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9540,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362300,"statfs":{"total":96636764160,"available":96626995200,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.45100000000000001,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45500000000000002,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51800000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":2.032,"15min":2.032},"last":0.255},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41099999999999998,"15min":0.42899999999999999},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60099999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.30299999999999999}]},{"osd":3,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.30399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":1.788,"15min":1.788},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.38800000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79600000000000004,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":1.002,"15min":1.002},"last":0.76200000000000001}]},{"osd":5,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85199999999999998,"5min":1.825,"15min":1.825},"last":0.748},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72699999999999998,"5min":1.75,"15min":1.75},"last":0.58499999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.84899999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.73899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9732,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94362108,"statfs":{"total":96636764160,"available":96626798592,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.19,"15min":0.19},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.314,"15min":0.311},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.245}]},{"osd":1,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.311,"15min":0.309},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.40699999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.28699999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.35199999999999998,"15min":0.34200000000000003},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.48499999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.34899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.219}]},{"osd":4,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.379},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.442}]},{"osd":5,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81000000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":1.294,"15min":1.294},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79000000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.48099999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.56200000000000006},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.46200000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81499999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.40000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10696,"kb_used_data":1280,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361144,"statfs":{"total":96636764160,"available":96625811456,"internally_reserved":0,"allocated":1310720,"data_stored":901765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34399999999999997,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":1.319,"15min":1.319},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38900000000000001,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.46999999999999997,"5min":1.915,"15min":1.915},"last":0.435}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.39900000000000002,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.45200000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.50700000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55500000000000005,"5min":1.278,"15min":1.278},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76200000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79400000000000004,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72399999999999998,"5min":1.371,"15min":1.371},"last":0.70699999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76400000000000001,"5min":1.857,"15min":1.857},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78800000000000003,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.71799999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.66000000000000003,"15min":0.67300000000000004},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80900000000000005,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.53100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9924,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361916,"statfs":{"total":96636764160,"available":96626601984,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.91800000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.41099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.008,"5min":1.073,"15min":1.073},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.47799999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71999999999999997,"5min":1.492,"15min":1.492},"last":0.54100000000000004}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51900000000000002}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.378},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39700000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67100000000000004,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.434}]}]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10760,"kb_used_data":1280,"kb_used_omap":0,"kb_used_meta":9472,"kb_avail":94361080,"statfs":{"total":96636764160,"available":96625745920,"internally_reserved":0,"allocated":1310720,"data_stored":901765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9699328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.85999999999999999,"15min":0.97199999999999998},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8069999999999999,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82699999999999996,"15min":0.95099999999999996},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.48099999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.90200000000000002},"min":{"1min":0.48599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.718,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.89400000000000002},"min":{"1min":0.42999999999999999,"5min":0.378,"15min":0.378},"max":{"1min":2.8559999999999999,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.90700000000000003},"min":{"1min":0.47899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.1139999999999999,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.377},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":4.891,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.89300000000000002,"15min":1.0089999999999999},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.0579999999999998,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75700000000000001,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.432}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.54900000000000004,"15min":0.51300000000000001},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.524,"5min":5.524,"15min":5.524},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":0.53100000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.52400000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.502},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.51100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:22:00.947 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.947 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.948 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.948 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.948 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.949 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.949 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.949 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.950 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2023-11-08T08:18:36.449394+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=18, tm_sec=36, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=8, tm_min=21, tm_sec=55, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T08:22:00.950 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-08T08:22:19.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:22:19.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:22:20.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T08:22:21.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.083+0000 7fc6b3c75700 1 Processor -- start 2023-11-08T08:22:21.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.083+0000 7fc6b3c75700 1 -- start start 2023-11-08T08:22:21.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b3c75700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06a270 0x7fc6ac06a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b3c75700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6ac070d90 0x7fc6ac0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b3c75700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ac151a50 con 0x7fc6ac06da40 2023-11-08T08:22:21.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b3c75700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac15f0c0 con 0x7fc6ac070d90 2023-11-08T08:22:21.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b3c75700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac15b440 con 0x7fc6ac06a270 2023-11-08T08:22:21.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b1210700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6ac070d90 0x7fc6ac0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b1210700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6ac070d90 0x7fc6ac0711e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51422/0 (socket says 172.21.15.116:51422) 2023-11-08T08:22:21.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b1210700 1 -- 172.21.15.116:0/1188898567 learned_addr learned my addr 172.21.15.116:0/1188898567 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:21.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.084+0000 7fc6b1a11700 1 --2- 172.21.15.116:0/1188898567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06a270 0x7fc6ac06a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b1210700 1 -- 172.21.15.116:0/1188898567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06a270 msgr2=0x7fc6ac06a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:21.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b1210700 1 --2- 172.21.15.116:0/1188898567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06a270 0x7fc6ac06a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b1210700 1 -- 172.21.15.116:0/1188898567 >> v1:172.21.15.129:6789/0 conn(0x7fc6ac06da40 legacy=0x7fc6ac06dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b1210700 1 -- 172.21.15.116:0/1188898567 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac06ab80 con 0x7fc6ac070d90 2023-11-08T08:22:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b1210700 1 --2- 172.21.15.116:0/1188898567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac0711e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098f0 tx=0x7fc6a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4834a20bce4c4d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b0a0f700 1 -- 172.21.15.116:0/1188898567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800b8a0 con 0x7fc6ac070d90 2023-11-08T08:22:21.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b0a0f700 1 -- 172.21.15.116:0/1188898567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a800ba00 con 0x7fc6ac070d90 2023-11-08T08:22:21.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b0a0f700 1 -- 172.21.15.116:0/1188898567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800bd90 con 0x7fc6ac070d90 2023-11-08T08:22:21.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b3c75700 1 -- 172.21.15.116:0/1188898567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 msgr2=0x7fc6ac0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/1188898567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac0711e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098f0 tx=0x7fc6a8009c00 comp rx=0 tx=0).stop 2023-11-08T08:22:21.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b3c75700 1 -- 172.21.15.116:0/1188898567 shutdown_connections 2023-11-08T08:22:21.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/1188898567 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06a270 0x7fc6ac06a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/1188898567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac0711e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.085+0000 7fc6b3c75700 1 -- 172.21.15.116:0/1188898567 >> 172.21.15.116:0/1188898567 conn(0x7fc6ac06c010 msgr2=0x7fc6ac0cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:21.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.086+0000 7fc6b3c75700 1 -- 172.21.15.116:0/1188898567 shutdown_connections 2023-11-08T08:22:21.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.086+0000 7fc6b3c75700 1 -- 172.21.15.116:0/1188898567 wait complete. 2023-11-08T08:22:21.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 Processor -- start 2023-11-08T08:22:21.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 -- start start 2023-11-08T08:22:21.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6ac06a270 0x7fc6ac150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06da40 0x7fc6ac0cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0b2b20 con 0x7fc6ac070d90 2023-11-08T08:22:21.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c5460 con 0x7fc6ac06a270 2023-11-08T08:22:21.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.087+0000 7fc6b3c75700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c4870 con 0x7fc6ac06da40 2023-11-08T08:22:21.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac15c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51438/0 (socket says 172.21.15.116:51438) 2023-11-08T08:22:21.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 -- 172.21.15.116:0/2529526765 learned_addr learned my addr 172.21.15.116:0/2529526765 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:21.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b1210700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06da40 0x7fc6ac0cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b1a11700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6ac06a270 0x7fc6ac150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 -- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06da40 msgr2=0x7fc6ac0cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06da40 0x7fc6ac0cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 -- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6ac06a270 msgr2=0x7fc6ac150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6ac06a270 0x7fc6ac150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 -- 172.21.15.116:0/2529526765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8009550 con 0x7fc6ac070d90 2023-11-08T08:22:21.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b2212700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac15c950 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400cc50 tx=0x7fc6a400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4003c00 con 0x7fc6ac070d90 2023-11-08T08:22:21.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a4003d60 con 0x7fc6ac070d90 2023-11-08T08:22:21.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac15adf0 con 0x7fc6ac070d90 2023-11-08T08:22:21.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400aab0 con 0x7fc6ac070d90 2023-11-08T08:22:21.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.088+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac071ac0 con 0x7fc6ac070d90 2023-11-08T08:22:21.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.089+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac15b0d0 con 0x7fc6ac070d90 2023-11-08T08:22:21.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.090+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc6a4010050 con 0x7fc6ac070d90 2023-11-08T08:22:21.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.090+0000 7fc6a27fc700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6900682f0 0x7fc69006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.090+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fc6a40857c0 con 0x7fc6ac070d90 2023-11-08T08:22:21.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.090+0000 7fc6b1a11700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6900682f0 0x7fc69006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.090+0000 7fc6b1a11700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6900682f0 0x7fc69006a7a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc69c006b20 tx=0x7fc69c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.094+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6a4052320 con 0x7fc6ac070d90 2023-11-08T08:22:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.269+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 --> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc6ac0c55d0 con 0x7fc6900682f0 2023-11-08T08:22:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.272+0000 7fc6a27fc700 1 -- 172.21.15.116:0/2529526765 <== mgr.4104 v2:172.21.15.116:6832/120856 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73058 (secure 0 0 0) 0x7fc6ac0c55d0 con 0x7fc6900682f0 2023-11-08T08:22:21.275 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:22:21.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6900682f0 msgr2=0x7fc69006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6900682f0 0x7fc69006a7a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc69c006b20 tx=0x7fc69c008040 comp rx=0 tx=0).stop 2023-11-08T08:22:21.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 msgr2=0x7fc6ac15c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac15c950 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400cc50 tx=0x7fc6a400cf60 comp rx=0 tx=0).stop 2023-11-08T08:22:21.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 shutdown_connections 2023-11-08T08:22:21.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ac070d90 0x7fc6ac15c950 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc6900682f0 0x7fc69006a7a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6ac06a270 0x7fc6ac150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 --2- 172.21.15.116:0/2529526765 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6ac06da40 0x7fc6ac0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 >> 172.21.15.116:0/2529526765 conn(0x7fc6ac06c010 msgr2=0x7fc6ac151120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:21.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 shutdown_connections 2023-11-08T08:22:21.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.274+0000 7fc6b3c75700 1 -- 172.21.15.116:0/2529526765 wait complete. 2023-11-08T08:22:21.280 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-08T08:22:21.287 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-11-08T08:22:20.400512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":79268,"kb_used_data":6756,"kb_used_omap":0,"kb_used_meta":72448,"kb_avail":754895452,"statfs":{"total":773094113280,"available":773012942848,"internally_reserved":0,"allocated":6918144,"data_stored":3670651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74186752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002339"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:21:58.816117+0000","last_change":"2023-11-08T08:21:58.816087+0000","last_active":"2023-11-08T08:21:58.816117+0000","last_peered":"2023-11-08T08:21:58.816117+0000","last_clean":"2023-11-08T08:21:58.816117+0000","last_became_active":"2023-11-08T08:18:37.485760+0000","last_became_peered":"2023-11-08T08:18:37.485760+0000","last_unstale":"2023-11-08T08:21:58.816117+0000","last_undegraded":"2023-11-08T08:21:58.816117+0000","last_fullsized":"2023-11-08T08:21:58.816117+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:58.815965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:58.815965+0000","last_clean_scrub_stamp":"2023-11-08T08:21:58.815965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:51:45.603180+0000","scrub_duration":0.0036970459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:06.197552+0000","last_change":"2023-11-08T08:22:06.197520+0000","last_active":"2023-11-08T08:22:06.197552+0000","last_peered":"2023-11-08T08:22:06.197552+0000","last_clean":"2023-11-08T08:22:06.197552+0000","last_became_active":"2023-11-08T08:18:37.461385+0000","last_became_peered":"2023-11-08T08:18:37.461385+0000","last_unstale":"2023-11-08T08:22:06.197552+0000","last_undegraded":"2023-11-08T08:22:06.197552+0000","last_fullsized":"2023-11-08T08:22:06.197552+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:22:06.197419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:22:06.197419+0000","last_clean_scrub_stamp":"2023-11-08T08:22:06.197419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:58:05.046303+0000","scrub_duration":0.0040781530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":60,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:02.597019+0000","last_change":"2023-11-08T08:22:02.596973+0000","last_active":"2023-11-08T08:22:02.597019+0000","last_peered":"2023-11-08T08:22:02.597019+0000","last_clean":"2023-11-08T08:22:02.597019+0000","last_became_active":"2023-11-08T08:18:37.460971+0000","last_became_peered":"2023-11-08T08:18:37.460971+0000","last_unstale":"2023-11-08T08:22:02.597019+0000","last_undegraded":"2023-11-08T08:22:02.597019+0000","last_fullsized":"2023-11-08T08:22:02.597019+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:22:02.596876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:22:02.596876+0000","last_clean_scrub_stamp":"2023-11-08T08:22:02.596876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:59:38.521654+0000","scrub_duration":0.0039746950000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"59'119","reported_seq":218,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:10.655898+0000","last_change":"2023-11-08T08:22:10.655869+0000","last_active":"2023-11-08T08:22:10.655898+0000","last_peered":"2023-11-08T08:22:10.655898+0000","last_clean":"2023-11-08T08:22:10.655898+0000","last_became_active":"2023-11-08T08:18:37.457897+0000","last_became_peered":"2023-11-08T08:18:37.457897+0000","last_unstale":"2023-11-08T08:22:10.655898+0000","last_undegraded":"2023-11-08T08:22:10.655898+0000","last_fullsized":"2023-11-08T08:22:10.655898+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'119","last_scrub_stamp":"2023-11-08T08:22:10.655768+0000","last_deep_scrub":"59'119","last_deep_scrub_stamp":"2023-11-08T08:22:10.655768+0000","last_clean_scrub_stamp":"2023-11-08T08:22:10.655768+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T08:28:55.932327+0000","scrub_duration":0.007003497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:02.141031+0000","last_change":"2023-11-08T08:22:02.140993+0000","last_active":"2023-11-08T08:22:02.141031+0000","last_peered":"2023-11-08T08:22:02.141031+0000","last_clean":"2023-11-08T08:22:02.141031+0000","last_became_active":"2023-11-08T08:18:37.461081+0000","last_became_peered":"2023-11-08T08:18:37.461081+0000","last_unstale":"2023-11-08T08:22:02.141031+0000","last_undegraded":"2023-11-08T08:22:02.141031+0000","last_fullsized":"2023-11-08T08:22:02.141031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:22:02.140885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:22:02.140885+0000","last_clean_scrub_stamp":"2023-11-08T08:22:02.140885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T13:58:00.989769+0000","scrub_duration":0.0040716290000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:21:59.469577+0000","last_change":"2023-11-08T08:21:59.469556+0000","last_active":"2023-11-08T08:21:59.469577+0000","last_peered":"2023-11-08T08:21:59.469577+0000","last_clean":"2023-11-08T08:21:59.469577+0000","last_became_active":"2023-11-08T08:18:37.460710+0000","last_became_peered":"2023-11-08T08:18:37.460710+0000","last_unstale":"2023-11-08T08:21:59.469577+0000","last_undegraded":"2023-11-08T08:21:59.469577+0000","last_fullsized":"2023-11-08T08:21:59.469577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:21:59.469475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:21:59.469475+0000","last_clean_scrub_stamp":"2023-11-08T08:21:59.469475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:51:46.256690+0000","scrub_duration":0.0037228000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:08.570441+0000","last_change":"2023-11-08T08:22:08.570389+0000","last_active":"2023-11-08T08:22:08.570441+0000","last_peered":"2023-11-08T08:22:08.570441+0000","last_clean":"2023-11-08T08:22:08.570441+0000","last_became_active":"2023-11-08T08:18:37.485662+0000","last_became_peered":"2023-11-08T08:18:37.485662+0000","last_unstale":"2023-11-08T08:22:08.570441+0000","last_undegraded":"2023-11-08T08:22:08.570441+0000","last_fullsized":"2023-11-08T08:22:08.570441+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:22:08.570169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:22:08.570169+0000","last_clean_scrub_stamp":"2023-11-08T08:22:08.570169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T18:36:16.425159+0000","scrub_duration":0.0039596170000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:02.490843+0000","last_change":"2023-11-08T08:22:02.490808+0000","last_active":"2023-11-08T08:22:02.490843+0000","last_peered":"2023-11-08T08:22:02.490843+0000","last_clean":"2023-11-08T08:22:02.490843+0000","last_became_active":"2023-11-08T08:18:37.485712+0000","last_became_peered":"2023-11-08T08:18:37.485712+0000","last_unstale":"2023-11-08T08:22:02.490843+0000","last_undegraded":"2023-11-08T08:22:02.490843+0000","last_fullsized":"2023-11-08T08:22:02.490843+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:22:02.490708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:22:02.490708+0000","last_clean_scrub_stamp":"2023-11-08T08:22:02.490708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T15:59:38.415486+0000","scrub_duration":0.0038125630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-11-08T08:22:00.496266+0000","last_change":"2023-11-08T08:22:00.496230+0000","last_active":"2023-11-08T08:22:00.496266+0000","last_peered":"2023-11-08T08:22:00.496266+0000","last_clean":"2023-11-08T08:22:00.496266+0000","last_became_active":"2023-11-08T08:18:37.485731+0000","last_became_peered":"2023-11-08T08:18:37.485731+0000","last_unstale":"2023-11-08T08:22:00.496266+0000","last_undegraded":"2023-11-08T08:22:00.496266+0000","last_fullsized":"2023-11-08T08:22:00.496266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T08:22:00.496073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T08:22:00.496073+0000","last_clean_scrub_stamp":"2023-11-08T08:22:00.496073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-09T14:00:21.969853+0000","scrub_duration":0.0038678760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9412,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362428,"statfs":{"total":96636764160,"available":96627126272,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":1.611,"15min":1.611},"last":0.51200000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.67100000000000004,"15min":0.72499999999999998},"min":{"1min":0.35699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.73899999999999999,"5min":5.8410000000000002,"15min":5.8410000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.68200000000000005,"15min":0.73099999999999998},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":5.9059999999999997,"15min":5.9059999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81299999999999994,"5min":5.8869999999999996,"15min":5.8869999999999996},"last":0.66200000000000003}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73499999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.621}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55900000000000005,"15min":0.60299999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59699999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57099999999999995,"15min":0.60099999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53400000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61499999999999999,"5min":0.872,"15min":0.872},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.42299999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69699999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.496},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70699999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9600,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362240,"statfs":{"total":96636764160,"available":96626933760,"internally_reserved":0,"allocated":712704,"data_stored":310136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.91200000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.115,"15min":1.115},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.44400000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":1.089,"15min":1.089},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.996,"15min":0.996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57999999999999996}]},{"osd":3,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.135},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.66000000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439}]},{"osd":5,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.67300000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46000000000000002}]},{"osd":7,"last update":"Wed Nov 8 08:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.501},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.747,"15min":0.747},"last":0.51400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9540,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362300,"statfs":{"total":96636764160,"available":96626995200,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.311,"15min":0.308},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.379,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.309},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":1.149,"15min":1.149},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.443,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.376}]},{"osd":3,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.50900000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48399999999999999,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.34999999999999998}]},{"osd":4,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.61099999999999999,"15min":0.56499999999999995},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70799999999999996,"5min":9.0380000000000003,"15min":9.0380000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.66200000000000003,"15min":0.63200000000000001},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.67700000000000005,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.70699999999999996}]},{"osd":5,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77600000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":0.56000000000000005}]},{"osd":6,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":8.5860000000000003,"15min":8.5860000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67300000000000004,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.72099999999999997}]},{"osd":7,"last update":"Wed Nov 8 08:21:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67600000000000005,"15min":0.629},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83199999999999996,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69299999999999995,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":9.0850000000000009,"15min":9.0850000000000009},"last":0.66200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9540,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362300,"statfs":{"total":96636764160,"available":96626995200,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.45100000000000001,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45500000000000002,"5min":5.1440000000000001,"15min":5.1440000000000001},"last":0.432},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51800000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.28199999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":2.032,"15min":2.032},"last":0.378},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41099999999999998,"15min":0.42899999999999999},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60099999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.30399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":1.788,"15min":1.788},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79600000000000004,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":1.002,"15min":1.002},"last":0.55700000000000005}]},{"osd":5,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85199999999999998,"5min":1.825,"15min":1.825},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72699999999999998,"5min":1.75,"15min":1.75},"last":0.64000000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.66100000000000003}]},{"osd":7,"last update":"Wed Nov 8 08:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.626},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9732,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94362108,"statfs":{"total":96636764160,"available":96626798592,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29699999999999999,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.19,"15min":0.19},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.314,"15min":0.311},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.33400000000000002}]},{"osd":1,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.311,"15min":0.309},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.40699999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.313},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.28699999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.38}]},{"osd":3,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.35199999999999998,"15min":0.34200000000000003},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.48499999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.372},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.34899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.40100000000000002}]},{"osd":4,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.75800000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56699999999999995}]},{"osd":5,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81000000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.57999999999999996}]},{"osd":6,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":1.294,"15min":1.294},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79000000000000004,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Wed Nov 8 08:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.56200000000000006},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77300000000000002,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.46200000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81499999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.61699999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10696,"kb_used_data":1280,"kb_used_omap":0,"kb_used_meta":9408,"kb_avail":94361144,"statfs":{"total":96636764160,"available":96625811456,"internally_reserved":0,"allocated":1310720,"data_stored":901765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9633792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34399999999999997,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":1.319,"15min":1.319},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38900000000000001,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.46999999999999997,"5min":1.915,"15min":1.915},"last":0.313}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.39900000000000002,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.45200000000000001,"5min":2.5070000000000001,"15min":2.5070000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.378}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.50700000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55500000000000005,"5min":1.278,"15min":1.278},"last":0.42199999999999999}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76200000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.68200000000000005,"15min":0.71699999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.78900000000000003}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79400000000000004,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72399999999999998,"5min":1.371,"15min":1.371},"last":0.61699999999999999}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76400000000000001,"5min":1.857,"15min":1.857},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78800000000000003,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.73399999999999999}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.68100000000000005},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.66000000000000003,"15min":0.67300000000000004},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80900000000000005,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361852,"statfs":{"total":96636764160,"available":96626536448,"internally_reserved":0,"allocated":716800,"data_stored":311397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.91800000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51500000000000001}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.497},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.41099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55400000000000005}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.008,"5min":1.073,"15min":1.073},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.47799999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71999999999999997,"5min":1.492,"15min":1.492},"last":0.48499999999999999}]},{"osd":5,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.438},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.34200000000000003}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36199999999999999}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67100000000000004,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.38500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10760,"kb_used_data":1280,"kb_used_omap":0,"kb_used_meta":9472,"kb_avail":94361080,"statfs":{"total":96636764160,"available":96625745920,"internally_reserved":0,"allocated":1310720,"data_stored":901765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9699328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.85999999999999999,"15min":0.97199999999999998},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8069999999999999,"5min":9.6790000000000003,"15min":9.6790000000000003},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82699999999999996,"15min":0.95099999999999996},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":9.7270000000000003,"15min":9.7270000000000003},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.90200000000000002},"min":{"1min":0.48599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.718,"5min":7.6239999999999997,"15min":7.6239999999999997},"last":0.745},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.89400000000000002},"min":{"1min":0.42999999999999999,"5min":0.378,"15min":0.378},"max":{"1min":2.8559999999999999,"5min":7.6079999999999997,"15min":7.6079999999999997},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.90700000000000003},"min":{"1min":0.47899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.1139999999999999,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.76400000000000001,"15min":0.80000000000000004},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.373},"max":{"1min":4.891,"5min":6.0209999999999999,"15min":6.0209999999999999},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.89300000000000002,"15min":1.0089999999999999},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.0579999999999998,"5min":9.7550000000000008,"15min":9.7550000000000008},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75700000000000001,"5min":3.5489999999999999,"15min":3.5489999999999999},"last":0.69999999999999996}]},{"osd":4,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.54900000000000004,"15min":0.51300000000000001},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.524,"5min":5.524,"15min":5.524},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":0.51200000000000001}]},{"osd":6,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed Nov 8 08:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T08:22:21.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-08T08:22:21.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 Processor -- start 2023-11-08T08:22:21.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 -- start start 2023-11-08T08:22:21.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc63815fe50 0x7fc63806d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6380b3b80 0x7fc638160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc638161370 con 0x7fc6380b27d0 2023-11-08T08:22:21.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc638160220 con 0x7fc6380b3b80 2023-11-08T08:22:21.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.411+0000 7fc63eff9700 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6380cf700 con 0x7fc63815fe50 2023-11-08T08:22:21.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6380b3b80 0x7fc638160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6380b3b80 0x7fc638160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51450/0 (socket says 172.21.15.116:51450) 2023-11-08T08:22:21.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 -- 172.21.15.116:0/4166485786 learned_addr learned my addr 172.21.15.116:0/4166485786 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:21.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc63cd95700 1 --2- 172.21.15.116:0/4166485786 >> v2:172.21.15.129:3300/0 conn(0x7fc63815fe50 0x7fc63806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 -- 172.21.15.116:0/4166485786 >> v2:172.21.15.129:3300/0 conn(0x7fc63815fe50 msgr2=0x7fc63806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:21.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 --2- 172.21.15.116:0/4166485786 >> v2:172.21.15.129:3300/0 conn(0x7fc63815fe50 0x7fc63806d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 -- 172.21.15.116:0/4166485786 >> v1:172.21.15.129:6789/0 conn(0x7fc6380b27d0 legacy=0x7fc6380cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 -- 172.21.15.116:0/4166485786 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380b3fd0 con 0x7fc6380b3b80 2023-11-08T08:22:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc637fff700 1 --2- 172.21.15.116:0/4166485786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6380b3b80 0x7fc638160ad0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc6380d1e30 tx=0x7fc624009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11524194995645d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc6377fe700 1 -- 172.21.15.116:0/4166485786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624004230 con 0x7fc6380b3b80 2023-11-08T08:22:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc6377fe700 1 -- 172.21.15.116:0/4166485786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc624004390 con 0x7fc6380b3b80 2023-11-08T08:22:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.412+0000 7fc6377fe700 1 -- 172.21.15.116:0/4166485786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6240036a0 con 0x7fc6380b3b80 2023-11-08T08:22:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 -- 172.21.15.116:0/4166485786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6380b3b80 msgr2=0x7fc638160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 --2- 172.21.15.116:0/4166485786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6380b3b80 0x7fc638160ad0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc6380d1e30 tx=0x7fc624009bd0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 -- 172.21.15.116:0/4166485786 shutdown_connections 2023-11-08T08:22:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 --2- 172.21.15.116:0/4166485786 >> v2:172.21.15.129:3300/0 conn(0x7fc63815fe50 0x7fc63806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 --2- 172.21.15.116:0/4166485786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6380b3b80 0x7fc638160ad0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 -- 172.21.15.116:0/4166485786 >> 172.21.15.116:0/4166485786 conn(0x7fc63806ea80 msgr2=0x7fc63806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:21.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 -- 172.21.15.116:0/4166485786 shutdown_connections 2023-11-08T08:22:21.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.413+0000 7fc63eff9700 1 -- 172.21.15.116:0/4166485786 wait complete. 2023-11-08T08:22:21.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.414+0000 7fc63eff9700 1 Processor -- start 2023-11-08T08:22:21.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.414+0000 7fc63eff9700 1 -- start start 2023-11-08T08:22:21.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63eff9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6380b27d0 0x7fc63815b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63eff9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6380b3b80 0x7fc6380c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63eff9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 0x7fc6380c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63eff9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380c74d0 con 0x7fc63815fe50 2023-11-08T08:22:21.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63eff9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63815b590 con 0x7fc6380b27d0 2023-11-08T08:22:21.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63eff9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63815ad10 con 0x7fc6380b3b80 2023-11-08T08:22:21.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 0x7fc6380c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 0x7fc6380c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51462/0 (socket says 172.21.15.116:51462) 2023-11-08T08:22:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 -- 172.21.15.116:0/1469506738 learned_addr learned my addr 172.21.15.116:0/1469506738 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63cd95700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6380b27d0 0x7fc63815b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc637fff700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6380b3b80 0x7fc6380c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 -- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6380b3b80 msgr2=0x7fc6380c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6380b3b80 0x7fc6380c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 -- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6380b27d0 msgr2=0x7fc63815b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6380b27d0 0x7fc63815b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 -- 172.21.15.116:0/1469506738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc624009550 con 0x7fc63815fe50 2023-11-08T08:22:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc63d596700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 0x7fc6380c5ee0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc63000b970 tx=0x7fc63000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63000c950 con 0x7fc63815fe50 2023-11-08T08:22:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.415+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63000cab0 con 0x7fc63815fe50 2023-11-08T08:22:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.416+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6380cf0b0 con 0x7fc63815fe50 2023-11-08T08:22:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.416+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63000ce60 con 0x7fc63815fe50 2023-11-08T08:22:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.416+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6380cedd0 con 0x7fc63815fe50 2023-11-08T08:22:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.416+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6380cf390 con 0x7fc63815fe50 2023-11-08T08:22:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.417+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc63001b030 con 0x7fc63815fe50 2023-11-08T08:22:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.417+0000 7fc6357fa700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc61c0682f0 0x7fc61c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.417+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fc630085440 con 0x7fc63815fe50 2023-11-08T08:22:21.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.417+0000 7fc63cd95700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc61c0682f0 0x7fc61c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.418+0000 7fc63cd95700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc61c0682f0 0x7fc61c06a7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009930 tx=0x7fc62c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.421+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc630051fa0 con 0x7fc63815fe50 2023-11-08T08:22:21.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.602+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fc6380c7e90 con 0x7fc63815fe50 2023-11-08T08:22:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.606+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc6300098b0 con 0x7fc63815fe50 2023-11-08T08:22:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.607+0000 7fc6357fa700 1 -- 172.21.15.116:0/1469506738 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) v1 ==== 123+0+0 (secure 0 0 0) 0x7fc6300555c0 con 0x7fc63815fe50 2023-11-08T08:22:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc61c0682f0 msgr2=0x7fc61c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc61c0682f0 0x7fc61c06a7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009930 tx=0x7fc62c008040 comp rx=0 tx=0).stop 2023-11-08T08:22:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 msgr2=0x7fc6380c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 0x7fc6380c5ee0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc63000b970 tx=0x7fc63000bd30 comp rx=0 tx=0).stop 2023-11-08T08:22:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 shutdown_connections 2023-11-08T08:22:21.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63815fe50 0x7fc6380c5ee0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fc61c0682f0 0x7fc61c06a7a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6380b27d0 0x7fc63815b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 --2- 172.21.15.116:0/1469506738 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc6380b3b80 0x7fc6380c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 >> 172.21.15.116:0/1469506738 conn(0x7fc63806ea80 msgr2=0x7fc6380b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:21.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 shutdown_connections 2023-11-08T08:22:21.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.609+0000 7fc63eff9700 1 -- 172.21.15.116:0/1469506738 wait complete. 2023-11-08T08:22:21.622 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-08T08:22:21.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-08T08:22:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.744+0000 7fe5a9260700 1 Processor -- start 2023-11-08T08:22:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.744+0000 7fe5a9260700 1 -- start start 2023-11-08T08:22:21.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a9260700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5a4142db0 0x7fe5a413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a9260700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40b1770 0x7fe5a40b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a9260700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40b2110 con 0x7fe5a40ce040 2023-11-08T08:22:21.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a9260700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a415e010 con 0x7fe5a4142db0 2023-11-08T08:22:21.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a9260700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a413fc90 con 0x7fe5a40b1770 2023-11-08T08:22:21.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a2d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe5a40ce040 0x7fe5a415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47882/0 (socket says 172.21.15.116:47882) 2023-11-08T08:22:21.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a2d9d700 1 -- 172.21.15.116:0/508726679 learned_addr learned my addr 172.21.15.116:0/508726679 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-08T08:22:21.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a259c700 1 --2- 172.21.15.116:0/508726679 >> v2:172.21.15.116:3300/0 conn(0x7fe5a4142db0 0x7fe5a413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a1d9b700 1 -- 172.21.15.116:0/508726679 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323975297 0 0) 0x7fe5a40b2110 con 0x7fe5a40ce040 2023-11-08T08:22:21.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a1d9b700 1 -- 172.21.15.116:0/508726679 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c0035b0 con 0x7fe5a40ce040 2023-11-08T08:22:21.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a259c700 1 -- 172.21.15.116:0/508726679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40b1770 msgr2=0x7fe5a40b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-08T08:22:21.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a259c700 1 --2- 172.21.15.116:0/508726679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40b1770 0x7fe5a40b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a259c700 1 -- 172.21.15.116:0/508726679 >> v1:172.21.15.116:6789/0 conn(0x7fe5a40ce040 legacy=0x7fe5a415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a259c700 1 -- 172.21.15.116:0/508726679 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a413f6d0 con 0x7fe5a4142db0 2023-11-08T08:22:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a259c700 1 --2- 172.21.15.116:0/508726679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a4142db0 0x7fe5a413ed70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe594009840 tx=0x7fe594009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d22be451369a4aa server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a1d9b700 1 -- 172.21.15.116:0/508726679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594011830 con 0x7fe5a4142db0 2023-11-08T08:22:21.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.745+0000 7fe5a1d9b700 1 -- 172.21.15.116:0/508726679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe594011990 con 0x7fe5a4142db0 2023-11-08T08:22:21.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a1d9b700 1 -- 172.21.15.116:0/508726679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594011d40 con 0x7fe5a4142db0 2023-11-08T08:22:21.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 -- 172.21.15.116:0/508726679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a4142db0 msgr2=0x7fe5a413ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 --2- 172.21.15.116:0/508726679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a4142db0 0x7fe5a413ed70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe594009840 tx=0x7fe594009b50 comp rx=0 tx=0).stop 2023-11-08T08:22:21.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 -- 172.21.15.116:0/508726679 shutdown_connections 2023-11-08T08:22:21.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 --2- 172.21.15.116:0/508726679 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40b1770 0x7fe5a40b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 --2- 172.21.15.116:0/508726679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a4142db0 0x7fe5a413ed70 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 -- 172.21.15.116:0/508726679 >> 172.21.15.116:0/508726679 conn(0x7fe5a406c010 msgr2=0x7fe5a40cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 -- 172.21.15.116:0/508726679 shutdown_connections 2023-11-08T08:22:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.746+0000 7fe5a9260700 1 -- 172.21.15.116:0/508726679 wait complete. 2023-11-08T08:22:21.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.747+0000 7fe5a9260700 1 Processor -- start 2023-11-08T08:22:21.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 -- start start 2023-11-08T08:22:21.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 0x7fe5a40c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40ce040 0x7fe5a4140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5a4142db0 0x7fe5a4141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c4150 con 0x7fe5a40b1770 2023-11-08T08:22:21.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41583e0 con 0x7fe5a4142db0 2023-11-08T08:22:21.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a9260700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c3820 con 0x7fe5a40ce040 2023-11-08T08:22:21.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 0x7fe5a40c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 0x7fe5a40c45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51470/0 (socket says 172.21.15.116:51470) 2023-11-08T08:22:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.748+0000 7fe5a2d9d700 1 -- 172.21.15.116:0/3042936301 learned_addr learned my addr 172.21.15.116:0/3042936301 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-08T08:22:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a2d9d700 1 -- 172.21.15.116:0/3042936301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40ce040 msgr2=0x7fe5a4140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a2d9d700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40ce040 0x7fe5a4140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a2d9d700 1 -- 172.21.15.116:0/3042936301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5a4142db0 msgr2=0x7fe5a4141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a2d9d700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5a4142db0 0x7fe5a4141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a2d9d700 1 -- 172.21.15.116:0/3042936301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940094a0 con 0x7fe5a40b1770 2023-11-08T08:22:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a2d9d700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 0x7fe5a40c45e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe59800c700 tx=0x7fe59800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598005d70 con 0x7fe5a40b1770 2023-11-08T08:22:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe5980070d0 con 0x7fe5a40b1770 2023-11-08T08:22:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a415d100 con 0x7fe5a40b1770 2023-11-08T08:22:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598007480 con 0x7fe5a40b1770 2023-11-08T08:22:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.749+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a413f9b0 con 0x7fe5a40b1770 2023-11-08T08:22:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.750+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a415d3e0 con 0x7fe5a40b1770 2023-11-08T08:22:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.750+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe598019030 con 0x7fe5a40b1770 2023-11-08T08:22:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.750+0000 7fe58b7fe700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5900682f0 0x7fe59006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-08T08:22:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.751+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5296+0+0 (secure 0 0 0) 0x7fe598084f20 con 0x7fe5a40b1770 2023-11-08T08:22:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.751+0000 7fe5a259c700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5900682f0 0x7fe59006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-08T08:22:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.751+0000 7fe5a259c700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5900682f0 0x7fe59006a7a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe5a406e770 tx=0x7fe594010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-08T08:22:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.755+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe598051a00 con 0x7fe5a40b1770 2023-11-08T08:22:21.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.977+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a40c4430 con 0x7fe5a40b1770 2023-11-08T08:22:21.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.977+0000 7fe58b7fe700 1 -- 172.21.15.116:0/3042936301 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fe598017df0 con 0x7fe5a40b1770 2023-11-08T08:22:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-08T08:22:21.979 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3463119896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3463119896},{"type":"v1","addr":"172.21.15.129:6834","nonce":3463119896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5968,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/4112571392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4112571392},{"type":"v1","addr":"172.21.15.129:6835","nonce":4112571392}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6204,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2238507316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2238507316},{"type":"v1","addr":"172.21.15.116:6836","nonce":2238507316}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3644503701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3644503701},{"type":"v1","addr":"172.21.15.116:6837","nonce":3644503701}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2023-11-08T08:22:21.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5900682f0 msgr2=0x7fe59006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5900682f0 0x7fe59006a7a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe5a406e770 tx=0x7fe594010000 comp rx=0 tx=0).stop 2023-11-08T08:22:21.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 msgr2=0x7fe5a40c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-08T08:22:21.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 0x7fe5a40c45e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe59800c700 tx=0x7fe59800cac0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 shutdown_connections 2023-11-08T08:22:21.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:6832/120856,v1:172.21.15.116:6833/120856] conn(0x7fe5900682f0 0x7fe59006a7a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5a4142db0 0x7fe5a4141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5a40b1770 0x7fe5a40c45e0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 --2- 172.21.15.116:0/3042936301 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe5a40ce040 0x7fe5a4140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-08T08:22:21.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 >> 172.21.15.116:0/3042936301 conn(0x7fe5a406c010 msgr2=0x7fe5a406ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-08T08:22:21.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 shutdown_connections 2023-11-08T08:22:21.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-08T08:22:21.979+0000 7fe5a9260700 1 -- 172.21.15.116:0/3042936301 wait complete. 2023-11-08T08:22:21.984 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2023-11-08T08:22:21.992 DEBUG:tasks.cephfs.filesystem:already dead... 2023-11-08T08:22:21.992 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-08T08:22:21.993 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-08T08:22:21.993 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.001 INFO:tasks.ceph.mds.a:Stopped 2023-11-08T08:22:22.002 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-08T08:22:22.002 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.050 INFO:tasks.ceph.mds.c:Stopped 2023-11-08T08:22:22.051 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-08T08:22:22.052 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.062 INFO:tasks.ceph.mds.b:Stopped 2023-11-08T08:22:22.063 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-08T08:22:22.063 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.113 INFO:tasks.ceph.mds.d:Stopped 2023-11-08T08:22:22.114 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-08T08:22:22.114 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-08T08:22:22.114 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.137 INFO:tasks.ceph.osd.0:Stopped 2023-11-08T08:22:22.138 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-08T08:22:22.138 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.203 INFO:tasks.ceph.osd.1:Stopped 2023-11-08T08:22:22.204 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-08T08:22:22.204 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.267 INFO:tasks.ceph.osd.2:Stopped 2023-11-08T08:22:22.267 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-08T08:22:22.268 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.330 INFO:tasks.ceph.osd.3:Stopped 2023-11-08T08:22:22.331 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-08T08:22:22.331 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.361 INFO:tasks.ceph.osd.4:Stopped 2023-11-08T08:22:22.362 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-08T08:22:22.362 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.435 INFO:tasks.ceph.osd.5:Stopped 2023-11-08T08:22:22.436 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-08T08:22:22.436 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.507 INFO:tasks.ceph.osd.6:Stopped 2023-11-08T08:22:22.507 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-08T08:22:22.508 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.581 INFO:tasks.ceph.osd.7:Stopped 2023-11-08T08:22:22.582 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-08T08:22:22.582 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-08T08:22:22.582 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.605 INFO:tasks.ceph.mgr.y:Stopped 2023-11-08T08:22:22.605 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-08T08:22:22.606 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.645 INFO:tasks.ceph.mgr.x:Stopped 2023-11-08T08:22:22.645 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-08T08:22:22.646 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-08T08:22:22.646 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.656 INFO:tasks.ceph.mon.a:Stopped 2023-11-08T08:22:22.657 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-08T08:22:22.657 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.696 INFO:tasks.ceph.mon.b:Stopped 2023-11-08T08:22:22.697 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-08T08:22:22.698 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T08:22:22.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-08T08:22:22.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-11-08T08:22:22.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-11-08T08:22:22.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-11-08T08:22:22.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-11-08T08:22:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-11-08T08:22:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-11-08T08:22:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-11-08T08:22:22.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-11-08T08:22:22.752 INFO:tasks.ceph.mon.c:Stopped 2023-11-08T08:22:22.752 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-08T08:22:22.753 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2023-11-08T08:22:22.785 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:22:22.786 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-08T08:22:23.013 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:22:23.014 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-08T08:22:23.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:22:23.142 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-08T08:22:23.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:22:23.209 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-08T08:22:23.270 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:22:23.271 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-08T08:22:23.443 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:22:23.443 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-08T08:22:23.526 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:22:23.527 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-08T08:22:23.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:22:23.572 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-08T08:22:23.640 INFO:tasks.ceph:Archiving mon data... 2023-11-08T08:22:23.642 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/data/mon.a.tgz 2023-11-08T08:22:23.643 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-11-08T08:22:23.662 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.lDiY8PDYTo 2023-11-08T08:22:23.662 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.lDiY8PDYTo 2023-11-08T08:22:23.805 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.lDiY8PDYTo 2023-11-08T08:22:23.892 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.lDiY8PDYTo is 361KB 2023-11-08T08:22:23.990 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.lDiY8PDYTo 2023-11-08T08:22:24.009 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/data/mon.b.tgz 2023-11-08T08:22:24.010 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-11-08T08:22:24.032 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.kvDXPaOLA0 2023-11-08T08:22:24.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.kvDXPaOLA0 2023-11-08T08:22:24.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.kvDXPaOLA0 2023-11-08T08:22:24.291 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.kvDXPaOLA0 is 474KB 2023-11-08T08:22:24.390 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.kvDXPaOLA0 2023-11-08T08:22:24.407 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/data/mon.c.tgz 2023-11-08T08:22:24.408 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2023-11-08T08:22:24.464 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.51fH0Nt2rA 2023-11-08T08:22:24.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.51fH0Nt2rA 2023-11-08T08:22:24.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.51fH0Nt2rA 2023-11-08T08:22:24.705 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.51fH0Nt2rA is 474KB 2023-11-08T08:22:24.803 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.51fH0Nt2rA 2023-11-08T08:22:24.820 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-08T08:22:24.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T08:22:24.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T08:22:24.887 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-08T08:22:24.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-08T08:22:24.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-08T08:22:24.918 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-08T08:22:24.954 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-08T08:22:24.960 INFO:tasks.ceph:Archiving crash dumps... 2023-11-08T08:22:24.962 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/remote/smithi116/crash 2023-11-08T08:22:24.963 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-08T08:22:24.992 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/remote/smithi129/crash 2023-11-08T08:22:24.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-08T08:22:25.027 INFO:tasks.ceph:Shutting down logrotate 2023-11-08T08:22:25.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:22:25.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-08T08:22:25.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-08T08:22:25.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-08T08:22:25.174 INFO:tasks.ceph:Compressing logs... 2023-11-08T08:22:25.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T08:22:25.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T08:22:26.688 INFO:tasks.ceph:Archiving logs... 2023-11-08T08:22:26.688 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/remote/smithi116/log 2023-11-08T08:22:26.690 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-08T08:22:27.200 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/remote/smithi129/log 2023-11-08T08:22:27.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-08T08:22:27.403 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-08T08:22:27.426 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-08T08:22:27.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-08T08:22:27.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-08T08:22:27.506 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-08T08:22:27.536 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-08T08:22:27.537 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T08:22:27.541 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T08:22:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:bash: ntpq: command not found 2023-11-08T08:22:27.559 INFO:teuthology.orchestra.run.smithi129.stderr:bash: ntpq: command not found 2023-11-08T08:22:27.561 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T08:22:27.561 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-11-08T08:22:27.562 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +1179us[+1179us] +/- 93ms 2023-11-08T08:22:27.562 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -1000us[-1005us] +/- 70ms 2023-11-08T08:22:27.563 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +102us[ +102us] +/- 65ms 2023-11-08T08:22:27.563 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:22:27.566 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T08:22:27.567 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2023-11-08T08:22:27.567 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +1189us[+1189us] +/- 93ms 2023-11-08T08:22:27.568 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -994us[ -937us] +/- 70ms 2023-11-08T08:22:27.568 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +112us[ +112us] +/- 65ms 2023-11-08T08:22:27.569 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T08:22:27.570 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-08T08:22:27.584 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-08T08:22:27.585 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-08T08:22:27.596 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-08T08:22:27.705 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-08T08:22:27.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-08T08:22:27.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-11-08T08:22:27.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-11-08T08:22:27.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-11-08T08:22:27.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-11-08T08:22:27.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-11-08T08:22:27.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-11-08T08:22:27.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-11-08T08:22:27.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-11-08T08:22:27.798 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T08:22:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:22:27.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T08:22:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:22:27.866 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-08T08:22:27.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-08T08:22:27.887 INFO:teuthology.task.internal:Duration was 1120.629244 seconds 2023-11-08T08:22:27.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-08T08:22:27.897 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-08T08:22:27.898 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-08T08:22:27.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-08T08:22:27.944 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T08:22:27.953 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T08:22:28.318 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-08T08:22:28.319 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2023-11-08T08:22:28.319 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-08T08:22:28.342 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2023-11-08T08:22:28.343 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-08T08:22:28.366 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-08T08:22:28.366 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T08:22:28.386 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T08:22:28.445 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-08T08:22:28.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-08T08:22:28.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-08T08:22:28.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-08T08:22:28.565 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-08T08:22:28.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-08T08:22:28.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-08T08:22:28.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-08T08:22:28.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-08T08:22:28.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-08T08:22:28.661 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2023-11-08T08:22:28.687 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2023-11-08T08:22:28.700 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-08T08:22:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:22:28.732 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-08T08:22:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T08:22:28.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-08T08:22:28.769 INFO:teuthology.task.internal:Transferring archived files... 2023-11-08T08:22:28.770 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/remote/smithi116 2023-11-08T08:22:28.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-08T08:22:28.810 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824/remote/smithi129 2023-11-08T08:22:28.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-08T08:22:28.849 INFO:teuthology.task.internal:Removing archive directory... 2023-11-08T08:22:28.849 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-08T08:22:28.852 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-08T08:22:28.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-08T08:22:28.913 INFO:teuthology.task.internal:Not uploading archives. 2023-11-08T08:22:28.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-08T08:22:28.964 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-08T08:22:28.965 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-08T08:22:28.967 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-08T08:22:28.982 INFO:teuthology.orchestra.run.smithi116.stdout: 526080 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 8 08:22 /home/ubuntu/cephtest 2023-11-08T08:22:28.985 INFO:teuthology.orchestra.run.smithi129.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 8 08:22 /home/ubuntu/cephtest 2023-11-08T08:22:28.986 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-08T08:22:29.029 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-08T08:22:29.055 INFO:teuthology.nuke:Checking targets against current locks 2023-11-08T08:22:29.083 DEBUG:teuthology.nuke:shortname: smithi116 2023-11-08T08:22:29.084 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T08:22:29.107 DEBUG:teuthology.nuke:shortname: smithi129 2023-11-08T08:22:29.107 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T08:22:29.111 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 07:55:31.821565', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmWrVyaQLvu7kH/xOHecmPZs0Wv4Vnk8ebDAlklT4ioAMKeecUfXKCBKWoONiWEWSMBVYZ7doSnrlZcsf0LePE='} 2023-11-08T08:22:29.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-08_07:51:45-fs-wip-neesingh-testing-54557-distro-default-smithi/7451824', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 07:55:31.823139', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP75L6TR03Z2dV6ScTobCTEsiEwEJDOeIRunLkcy5G1avJ+xdQ9cy9Bds1buAVhxI7AAzWZJ//guXa7HXaHSMi0='} 2023-11-08T08:22:29.164 INFO:teuthology.orchestra.console.smithi116:Power off 2023-11-08T08:22:29.165 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T08:22:29.190 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2023-11-08T08:22:29.191 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:29.208 INFO:teuthology.orchestra.console.smithi129:Power off 2023-11-08T08:22:29.209 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T08:22:29.222 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-11-08T08:22:29.238 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2023-11-08T08:22:29.238 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:29.266 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-11-08T08:22:33.223 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:33.267 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:33.333 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-11-08T08:22:33.378 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-11-08T08:22:37.336 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:37.379 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:37.448 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2023-11-08T08:22:37.489 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2023-11-08T08:22:37.549 INFO:teuthology.orchestra.console.smithi116:Power off completed 2023-11-08T08:22:41.490 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T08:22:41.601 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2023-11-08T08:22:41.702 INFO:teuthology.orchestra.console.smithi129:Power off completed 2023-11-08T08:22:41.807 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} duration: 1120.6292436122894 failure_reason: 'Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)' flavor: default owner: scheduled_neesingh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6ffcb431f28d44b3b00d0c6363878d22 status: fail success: false 2023-11-08T08:22:41.808 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T08:22:41.892 INFO:teuthology.run:FAIL